Virtualization and Cloud executives share their predictions for 2013. Read them in this VMblog.com series exclusive.
Contributed article by Jean-François Huard, Ph.D., CTO, Netuitive
How the Cloud and Big Data Trends Will Impact APM Analytics
As we approach the New Year
Data" is everywhere - including traditional IT operations management and
application performance management (APM). What does it mean and what is the impact? Just as Cloud Computing bursted onto the
scene a few years ago, it depends on whom you ask.
Traditionally, in the Business
Intelligence (BI) world, Big Data included analyzing historical business data
from large data warehouse with the purpose of identifying long-term trends that
could be leveraged in consumer business strategies. In recent years, Big Data has been a term
talked about in the IT industry as an application of technology to attack
extremely large, unstructured data sets that can reside both within and outside
of an organization. If you look at a recent
definition of Big Data, it is a term applied to data sets whose size has grown
beyond the capability of commonly used software tools to capture, manage and
analyze within a tolerable period of times for different use cases.
Application Performance Management (APM)
is an extremely relevant use case and has a developing "Big Data" problem. Several
factors are contributing to the explosive growth and type of data that must be analyzed
and/or correlated in application performance monitoring and business service
First, the number of components that
make up today's mission critical applications has exploded. Instead of
hundreds of servers for an application, nowadays, because of virtualization, you
can easily be talking about thousands of virtual servers and objects for web
Secondly, the diversity of data that
people want to analyze to provide a holistic perspective has increased drastically. It is no longer good enough to simply
understand traditional IT infrastructure performance based on server operating
system, network traffic, and storage capacity.
Application Performance analysis is now based on the relationships of IT
infrastructure components, application performance metrics from applications
and application servers, business activity monitors (BAM) data, customer
experience monitors (CEM) and Real-User Monitoring (RUM). In addition to the
aggregated transactional data, there are new systems that capture transactions'
actual path encompassing the entire application stack.
Finally, the requirements for analysis speed
and data granularity have also increased significantly. Mission critical application performance now
requires real-time or near real-time data analysis. When we were doing server availability and
performance monitoring 10 years ago, it was the norm to collect and analyze data
every 15 minutes. Today, this has evolved to data analysis every 5
minutes or less with sub-minute data collection where all transaction paths are
collected for data analysis. When
mapped out, it's easy to see the enormous growth particularly when you look at APM
related storage requirements that are quickly growing from gigabytes to
terabytes and tomorrow petabytes.
All this data requires extremely complex
analysis and correlation in order to truly understand performance of critical
applications. One of Netuitive's large
enterprise customers reported that it monitors and correlates a billion
infrastructure and application data points and business metrics daily as part
of its global service delivery. This is
what I am referring to as APM-generated
In addition to the shear number of data points, IT operators are expected to
provide real-time analysis to the business and long-term storage for
post-mortem analysis, capacity planning and compliance.
As we approach 2013 this is where APM
and Big Data meet The Cloud. The cloud can deliver cheaper and more flexible
storage and computing power crucial to analytics for Big Data. It also has the
capability to be much more elastic for your APM data storage and analytics needs.
Organizations can actually think about storing years of collected and
aggregated APM data for compliance and analysis purposes without the cost being
But what does this mean to vendors in
the APM space?
First of all, the analytics platform for
APM data has to evolve to be able to process the growing number of different
data sources across business, customer experience, applications and IT domains.
Netuitive's "Open" analytics platform is engineered to address virtually any
data source in real-time.
Secondly, data storage and access time
will be critical even as APM data volumes continue to explode, so not only does
the technology need to be able to run in the cloud, but the traditional
pull-based data collection architecture has to evolve into a push based model
with an horizontally scalable computing and storage architecture in order to
become virtually limitless in terms of scalability. This is critical for
larger organizations as "real" time no longer means analysis every 5 to 15
minutes, but sub-minute analytics.
because storage and computing costs should not significantly exceed the cost of
analytics software for a solution to be viable, Netuitive is advancing its
product architecture to leverage NoSQL columnar data store as a replacement to
traditional database. Netuitive is also experimenting with a SaaS model for
long-term time series data store and running its analytics software in the
our R&D challenges are complex, the goal is simple: provide APM Analytics
that matters by enabling our enterprise customers to process billions of infrastructure,
application, and business metrics from hundreds of thousands of managed
elements at 10x less cost than existing infrastructures.
I look forward to reporting on our progress.
About the Author
Jean-Francois Huard, Ph.D., CTO
Jean-François (JF) Huard is Chief Technical Officer and Vice
President of Research and Development at Netuitive, Inc. In this role he is responsible
for leading the company's vision and technology innovation effort. Previously,
he worked in network control and management focusing on optimal flow control,
decision-theoretic troubleshooting and game-theoretic bandwidth trading. His
current interests focus on IT analytics, cloud management and big data. Jean-François
received his Ph. D. (EE) from Columbia University, New York.