Welcome to Virtualization and Beyond

Cleaning Pesky Virtualization Management Challenges Out of the Garage

Written by Chris Paap, Technical Product Manager, SolarWinds

As part of a garage cleanup effort I had valiantly procrastinated my way out of for the better part of a year, I came across boxes containing remnants of my IT past. Among the pictures and plaques was an old binder I used to keep for my team containing handwritten notes and build guides that were the result of some hard earned knowledge usually gained in the early morning hours after a marathon troubleshooting session. The idea of the binder was that the team could update and track issues and log changes that could then be applied to our SOWs and team procedures. Aside from providing a trip down memory lane for some admittedly cringe-worthy experiences, I noticed how the issues we addressed in that binder are still some of the most common I come across as I work with virtualization administrators to resolve issues in their virtual environments as part of my current role at SolarWinds.

As such, I thought it may be useful to address some of them here, too.

Resource Utilization

No matter if you are managing a legacy virtual environment or a greenfield installation with new hardware, it seems there are never enough virtual resources (RAM, CPU, storage and networking).  More specifically, there are never enough virtual resources where and when you need them. This is where proactively right-sizing your environment pays dividends. Also key is identifying orphaned virtual disks wasting terabytes of data, or VMs that were over-allocated with too much RAM or multiple vCPUs where one would do. The latter issue of over-allocating vCPUs is a particularly common problem in environments where a template was used with little regard to resource utilization, resulting in performance degradation due to vCPU contention. 

Capacity Planning

Proper capacity planning can help identify future workloads, which is essential in determining what the potential top end limit of the current environment will be, but also determine how much more is needed to accommodate future growth. If your experience is anything like mine, budget cycles for additional equipment never come fast enough, and no matter how much you prepare, there are always unexpected projects or resource needs-for example, that pesky application workload that all of a sudden goes through the roof. Knowing when your resources are going to run out (resource depletion) provides the deadline to get remediation efforts underway or adequate reasoning for new purchases.  Capacity planning with simulated workloads can be accomplished by with a number of helpful tools, but even if you don't have such a tool, there is still value in identifying trends around resource utilization with whatever monitoring solution you are using combined with your knowledge of the environment to ballpark your resource depletion.

Configuration Drift

The larger an environment gets, the harder it becomes to track changes to the configuration of individual hosts causing configuration drift. The resulting configuration discrepancies may only become apparent during critical events where a host does not properly execute a high availability action because of port group mismatches caused by too many administrator hands in the pot. Ideally, you should have a virtualization monitoring solution that alerts you when a configuration change occurs, which can do wonders to prevent configuration drift and also keep your hosts in compliance. However, there are a couple other steps you can take to avoid this proverbial self-inflected IT wound. First, leverage proper virtualization administrator roles, which can help prevent inadvertent configuration changes from other administrators by locking down what accounts have access to upper level administration activities. And remember, leveraging the least privileged administrative models prevents errant changes. Second, be very mindful that updating drivers or replacing hardware can also wipe out configuration variables set to provide the best performance. So, plan ahead and ensure you have configuration backups in the event something gets inadvertently wiped away.

Monitoring

Monitoring should be the backbone of any good, ongoing IT strategy, which is why it's already come up prior to now. But it's so important that it get its own section, too. Monitoring your virtual environment must go beyond simply knowing the up or down status of your VMs and hosts. Whether using native tools or more robust third-party monitoring platforms, monitoring not only helps you ensure your environment is operating at peak performance now, but also helps you ensure it will scale by capturing all the important environmental metrics needed to make wise decisions. And remember, when establishing your monitoring practices, ensure that you set your alert thresholds to warn you well ahead of a critical points so steps can be taken to remediate potential issues before resulting in unexpected downtime. Also, set up your reporting to focus on critical, immediate needs and issues as well as longer tail reporting to help you identify trends in your environment.

Administrator Education

It goes without saying that an environment is only as good as the administrator who runs it. A very common scenario is that of the accidental administrator. It usually starts with administrator who wears many hats building a small virtual environment where capacity and performance were not of critical concern. With time, however, the utilization of the virtual environment grew, and as it did, so too did its importance and visibility within the organization. In such cases, the administrator is often forced to learn on the fly. Afar better approach, however, is for the administrator to gain the support of the organization to acquire the training and tools necessary to ensure the environment work well now and into the future. I realize, however, for some, this simply will never become reality. Thus, in lieu of paid training, those of you who fall into this category should make use of blogs and IT communities, such as the SolarWinds THWACK community, where virtualization experts and the community at large can provide much needed education and assistance. Also, look to leverage the hands-on labs often provided by hypervisor vendors.

Conclusion

In closing, don't let your virtual environment become like my garage (and perhaps yours, too)-neglected for far too long, making the task of getting it in check much more difficult then it needs to be. Improving your resource utilization and capacity planning, preventing configuration drift, leveraging good monitoring and educating yourself will pay dividends in the long run.

##

Read more articles from the Virtualization and Beyond Series.   

About the Author