Industry executives and experts share their predictions for 2022. Read them in this 14th annual VMblog.com series exclusive.
The Maturity of Observability
By Nick Heudecker, Senior Director of Market Strategy & Competitive Intelligence, Cribl
Investors are rapidly funneling money into the hot market of observability, with both promising start-ups and long-standing companies jumping into the space. But with the various pillars – observability data, observability pipelines, observability infrastructure – not to mention the conflation with traditional “monitoring,” the waters are already starting to muddy.
This year, I expect the conversation around observability will become more nuanced and sophisticated, as the market continues to mature and enterprises start seeing the true value observability can bring across teams.
At Cribl, we spend all year analyzing the market, exploring trends, and seeing what’s on the horizon. Based on our own analysis and what we’re hearing from customers, partners, and prospects, here are our top three observability predictions for 2022.
Prediction #1: Observability Moves In House
There are dozens, if not hundreds, of products touting complex machine learning models aiming to capitalize on your observability data. Some describe themselves as delivering AIOps to overworked operations and security teams, while others deliver pre-packaged models and APIs for these teams to use and implement on their own.
These products have had mixed successes. Many of these products assume they’re the only tool in the stack, making them hard to integrate with other tools, or with adjacent business processes. Another problem is relevance. The models and approaches used by many vendors are generic and may not be applicable to the specific problems operations teams face.
Over the next year, operations teams will shift away from these monolithic, generic automation solutions and towards more home-grown implementations built to solve the most pressing security and operational challenges. Rather than focusing on a single tool and data silo, they’ll build these tools with a mix of technologies accessing data from across the enterprise.
Prediction #2: Security Teams Drive Observability Maturity
Much of the conversation around observability in 2021 targeted developers, with the view that developers are also the operators of code. While this view is popular in Silicon Valley, the reality is often quite different outside the Bay Area. Developers are expensive, and having them spend time on operational tasks they’re not experts in is a waste of time and effort. Instead of developers driving the observability discussion, we’ll see cybersecurity teams taking the driver’s seat and leading this transformation in their companies.
Security teams are heavy users of monitoring already, deploying a range of tools to uncover known threats. But these tools fall short in three ways. First, they take a one-size-fits-all to the data they ingest. They ignore varying levels of data quality and value, treating all data as the same. Second, current pricing models make broad security monitoring cost prohibitive. Ingest-based pricing penalizes users for every byte ingested, while workload-based pricing penalizes users for every search they run. These limitations hamper investigations and slow remediation. Finally, no tool or platform owns all of the data, resulting in a fragmented data picture. An observability-based security architecture must take into account all of these fragments and weave them into a coherent picture.
When it comes to security use cases, security teams are realizing that all data is in scope, not only the traditional data sources. Security-focused observability shifts the conversation from what teams already know to look for to uncovering new threats and attack patterns on the horizon while also remediating existing breaches. For security professionals, observability offers benefits around things such as: routing data to multiple destinations for advanced analysis across a range of tools; enriching data with additional context, allowing teams to ask better questions and support the adoption of AIOps; and conducting faster and more accurate post-mortems on security events.
As security teams explore how observability practices benefit them, they’ll demand more accessibility to observability data and better tools to manage it. These needs will become more acute as teams migrate away from legacy platforms to modern solutions.
Prediction #3: Observability Lakes Become the New System of Discovery
Today’s monitoring and observability environments are fragmented across dozens of different tools and data silos. These environments haven’t kept pace with the realities many overworked infrastructure, operations, and security teams deal with, like ephemeral infrastructure, highly distributed applications, and complex application architectures. Additionally, there is an operational imperative for pervasive instrumentation in modern applications. Fully instrumented applications can generate terabytes of data per day, creating acute challenges around observability data management.
An observability lake centralizes observability data for disparate teams and offers the features and functionality they need to understand their rapidly evolving environments. Unlike traditional data lakes, with their focus on data processing and analytics ranging from self-servicing canned reports to machine learning use cases, observability lakes optimize for the experience of the user. SREs, SecOps, and DevOps people aren’t SQL users. They understand search and regular expressions. Observability lakes must offer robust search capabilities coupled with powerful time-based features like binning, aggregations, and ranges. Those features must work across a range of different data formats and types.
Final word
The category of observability is currently at the peak of Gartner’s Hype Cycle. It’s gaining momentum among vendors and analysts, but to many end users, it’s still a mystery. I’m optimistic that 2022 will be a breakout year for observability as businesses realize the power of observability to bring immense value to IT, operations, security, and business stakeholders.
##
ABOUT THE AUTHOR
Nick Heudecker is the Senior Director of Market Strategy & Competitive Intelligence at Cribl. Prior to joining Cribl, he spent over seven years as an industry analyst at Gartner, covering the data and analytics market. With over twenty years of experience, he has led engineering and product teams across multiple successful startups in the media and advertising industries.
