If you aren't familiar with vSphere Log Insight, and you're running a VMware vSphere infrastructure... you should get to know this product. It is VMware's solution for log management and analytics of dynamic hybrid cloud environments. The product analyzes large amounts of unstructured machine generated log data, and provides deep, enterprise-wide visibility. It also provides interactive, real-time search and analytics of that data in a meaningful way so that IT users can identify and analyze the data, then use that intelligence to proactively enable service levels and operational efficiencies across dynamic hybrid cloud environments.

VMware recently launched vCenter Log Insight 2.0, and included a number of new and interesting features worth checking out.

But I wanted to dig in a little deeper and find out more, so I spoke with Bill Roth, the group product line marketing manager at VMware.

VMblog:  With 2.0, VMware has added a number of new features.  What's the uptake been like on vCenter Log Insight so far? 

Bill Roth:  The uptake has been exceptional. In addition to the web evaluations, customers have been trying our new automated Hands On Lab, http://bit.ly/log-insight-lab1, where they can try the product without the need to install it.

We have been very promotional in the first nine months of the product's life, and will continue to make this product available to as many customers as possible. To that end, vCenter Log Insight 2.0 has a number of features that will make it more attractive to users, including a new, sleek user interface, and more powerful analytic visualizations.

VMblog:  What has VMware been hearing from consumers, any feedback you can share?

Roth:  I can tell you about a couple of customers I have been working with. There is a banking customer who is monitoring 400 Hosts, 12,000 virtual machines (VMs). They keep about 40 GB/day in logs, and it cut their data center problem solving time by 50 percent.

There is another consumer products company with 30 hosts and 300 VMs total, and some of the VMs were getting trashed nightly. It was a persistent problem, but they were able to find the problem quickly after installing vCenter Log Insight. They determined it was a backup software configuration error.

VMblog:  How close to real-time is the data collection and analysis?  And how fast is it?

Roth:  Within seconds based on the logs that are generated from the source when any type of change occurs, the transit time to vCenter Log Insight, the filtering time and the time it takes to represent within the user defined custom dashboards. There are no alternative technologies that collect and analyze faster than with the use of the log files. The key enabler is how quickly a log file is generated and sent from the source, based upon the change activity within this source. 

VMblog:  And is there a simple way to tie this technology and its analytics into provisioning engines to enable automated application service level management?

Roth:  Provisioning engines today for automating cloud infrastructure based customer and application onboarding are typically several interconnected domain and portal tools with workflow managers such as vCenter Orchestrator. The workflow manager offers the greatest single point of integration as it can provide information regarding the services being deployed and the unique identity of these services to the operations tools, including vCenter Log Insight.  As each customer deployment of these provisioning engines are different there is no predefined set of workflow for this, yet with step by step workflow management, vCenter Log Insight can be integrated with the collection and filtering of services deployed.

VMblog:  How many different device types can it identify, and what about capturing and analyzing log information?  Is it limited to specific hardware vendors?  Or does it work across generic devices?

Roth:  Almost all modern data center operating systems, whether the device is physical or virtual provide log information, based upon any device change activity. Therefore, vCenter Log Insight can integrate across all data center products. Subject matter experts that understand the change events, and what these change events mean operationally, are required for customizing vCenter Log Insight for their device class or specific device type. For some device classes such as networking or security, there are well known log formats that many vendors follow, although not compelled to based upon industry norms and what admins are accustomed to. This helps in developing more generic device class content packs.

VMblog:  I love the idea behind these Content Packs.  Who, if anyone, is helping to create them and what's the response rate been like on these from a development and consumption basis? 

Roth:  vCenter Log Insight offers public guidelines on how to develop a content pack.  Anyone familiar with log files, relative to their operations needs can create these content packs, including technology partners, third party integrators, consulting services and/or customers. The uptake currently in creating these content packs has been within the technology, third party integrators, and consulting services.

The primary challenge in creating these content packs is around the identification of the operation problems the vendor or the consulting company is looking to resolve, and finding the right content owners that have enough familiarity with the log files (subject matter experts) for determining the filters, dashboards and alerts. Many of these content experts are within the technology partner(s) support organization. These support managers are not the primary alliance interface to VMware. Engaging the VMware ecosystem engineering and alliance organization will help deepen the technology partner efforts both in identifying these key support resources as well as working with them in developing the content pack use cases. These efforts will drive both the development and consumption of content packs in the second half of 2014 for VMware.

VMblog:  VMware is known for its community of users.  Can you tell us, how has community effort been with this product thus far? 

Roth:  The initial uptake within the data center virtualization community has been good per the 14 content packs offered. VMware is now taking this a step further, per its work in the past six months to offer a more structured program through the VMware ecosystem alliance and engineering organization. This organization will offer consulting and support, in conjunction with the Cloud Management Business Unit, to further work with the community to develop vCenter Log Insight content packs.

VMblog:  And what about third-party companies? 

Roth:  Both physical and virtual data center infrastructure third-party technology providers as well as customers are in need of modern day operation management tools that are:

a) low cost to develop/consume; 

b) work ubiquitously across a diverse set of technologies;

c) can handle the volume of operations data generated by thousands of devices; and,

d) can in real-time flag critical outage and change conditions that have impact to their workloads.

The ability to collect log information in real-time, at high volume rates, has become this ubiquitous source of operations data; and is relatively easy to integrate based on key attribute pair filtering with custom dashboards and alerts. Third parties are interested in integrating with vCenter Log Insight as this product offers all of these capabilities. The most compelling is the low price point of this product relative to others within the market, and the ease in which third parties can develop content packs without the need for any custom code, scripts or source code. 

VMblog:  Are there any content packs that stand out or that you have received feedback on as being a must have?  If so, why?

Roth:  Clearly collecting, analyzing, correlating of unstructured log data, specific to virtualization infrastructure is of strong interest and need to virtual administrators. This includes log data related to vSphere (ESXi) and View (VDI). These two content packs stand out, especially in larger infrastructures in which the intelligent collection and filtering of log data is the only way to sort through millions of log files per month. Virtual administrators are now asking to extend vCenter Log Insight into other related areas including storage, network, and physical server information. These extensions help them to understand outages, configuration changes, etc., that directly impact the virtualized workloads they are responsible for.

VMblog:  Any other useful or interesting ways of using this technology surface yet?  What can people expect?

Roth:  The most interesting or compelling use cases are based around time to market, low bar to integrate, and new entrants into a particular market segment of the data center. This is based on the following:

The most compelling aspect of log management is that all data center technologies provide this information source. The downside up until recently is that log files are highly unstructured and difficult to scale across thousands of sources. Because of this, log files are a hard source to develop management tools with, and as a result many vendors have had to develop more complex and sophisticated management protocols and messages including SNMP, REST, XML, NetConf, and others too long to mention here. This adds cost and complexity to their development efforts as well as taking a longer time to develop as these protocols require development for every feature they wish to be managed.

With the advent of collecting and filtering log messages in near real-time (within seconds), and to provide meaningful GUI representations of this unstructured data, log files have become a fast way to develop management capabilities and get the products they support to market. As a result, we are seeing many new entrants to the data center market develop to vCenter Log Insight as a faster, more nimble way to provide operations management. ExtraHop and NetFlow Logic are two partners already on VMware Solution Exchange that have taken this route. There are multiple partners within this category that are in development as well. Moreover, many traditional partners who have very mature logs, as well as the more sophisticated management protocols, are integrating with vCenter Log Insight as they also see fast route to market in offering integrated management with VMware with many of their new data center infrastructure features.

##

Once again, a special thanks to Bill Roth, group product line marketing manager at VMware, for taking time out to answer a few questions about one of the company's latest products, vCenter Log Insight 2.0.