Industry executives and experts share their predictions for 2024.  Read them in this 16th annual VMblog.com series exclusive.

Three Observability Trends That Will Resonate in 2024 – And What To Do About Them

By Asaf Yigal, Co-Founder and CTO, Logz.io

For organizations that rely on their customers' digital experiences to meet their business objectives, keeping applications and infrastructure running smoothly directly impacts the bottom line. That's why observability-visibility into the IT system's health and performance-is essential. Unfortunately, despite its mission-critical role in modern enterprises, observability has been defined by crippling complexity and high costs. In 2024, complexity and cost issues will continue to plague the industry, with three specific challenges rising to the top as particularly problematic. Here are those three challenges and what you can do to address them.

In our annual DevOps Pulse survey, we have witnessed the gradual increase of the Mean Time To Recovery (MTTR) for production issues year over year. In the 2021 survey, 47% of respondents stated that it took multiple hours on average to resolve production issues. This rose to 64% of respondents in 2022, and a stunning 73% in 2023.

A major impetus behind this growing challenge is expanding system complexity, especially due to the adoption of Kubernetes and cloud-native technologies and practices. Technologies such as Kubernetes generate abundant and complex data, making it difficult to monitor and troubleshoot. In fact, these technologies were cited by 46% of this year's survey respondents as the most difficult obstacle for organizations to gain full observability of their environment.

To address this challenge, enterprises will need ways to troubleshoot faster, and AI can help. For example, AI can help reduce the volume and complexity of our staggering data pipelines - before we even begin the day-to-day analysis - to help us focus on and pay for only the data that matters most.

AI can also inform the prioritization of alerts, so that our human analysts, regardless of responsibility, spend their time tracking down the performance issues or threats that represent the biggest risk to business viability. In addition, once an issue is prioritized, crowdsourced AI, ChatGPT, and AI modeling of successful resolutions can help engineers take the right actions more efficiently.

As you explore options for observability solutions, look for those that incorporate the power of AI to hone in and address troubleshooting issues with speed and precision.

One problem with running cloud-native applications on multi-cloud, multi-region architectures is that they generate enormous amounts of observability data-and increasingly so as organizations scale. With the volume of observability data exploding, most organizations aren't equipped to handle the budget-breaking costs that ensue.

We can beat this observability bloat by investing in smart data collection and data hygiene. Many organizations just collect everything they can get their hands on. This costs a lot of money, creates noisy clutter in the environment, and hampers the ability to search and gain insights.

Instead, observability solutions should help organizations filter out unneeded data to reduce costs and also help customers move their less valued data into far less expensive storage repositories without losing the ability to actively query it when needed.

An emerging trend in 2023-one that is sure to grow in 2024-is the need for shared visibility into key enabling apps and IT infrastructure technologies from both an operational and security standpoint. Kubernetes is a primary example. No matter what model or teams you support internally, there's a shared interest in the performance and security of technologies like Kubernetes that are so fundamental to modern apps and infrastructure.

Regardless of whether security and observability will converge in terms of organizational structure and roles and responsibilities of teams, there does seem to be a big advantage to adding security capabilities to the traditional three pillars of logs, metrics and traces for observability.

Whether you have "ops"-type teams that can act on that data themselves or use it as a better informed stream of data to channel to their dedicated security teams, the reality is that cloud apps and infrastructure are so complex and fast moving, security has to be part of the picture for everyone involved.

We know for sure that there is a huge benefit in bringing together the relevant data, either to be actioned centrally, say in a smaller shop with only a few people responsible for DevSecOps, or to be communicated across teams in a larger organization with multiple groups spanning the entire landscape. So while only the most mature organizations are starting to converge their approach to some of these processes, we do see a growing appetite for shared visibility, workflows, and context.

Companies like mine see this as both a challenge and opportunity, and we're actively exploring a variety of practices that will bring this to fruition-from employing shared observability and security in a single platform to building tighter integrations through automated workflows, when possible.

Looking ahead to 2024, the good news for companies like mine in the observability business is that observability is more important than ever before. However, as the complexity of applications and infrastructure increase, so do our challenges. Our strategy is to keep our focus on what provides value for our customers...and in 2024 the key themes will be troubleshooting faster, reducing costs, and providing "mutual visibility" to support both operations and security objectives.

##

ABOUT THE AUTHOR

Asaf Yigal is the CTO and co-founder of Logz.io, provider of the world's most innovative, easy-to-use observability platform. Unlike traditional solutions that continuously increase complexity and costs, Logz.io Open 360^TM enables engineering teams to replace siloed data with a single platform purpose-built to deliver rapid insights and end-to-end cost efficiency.