Era Software recently announced the findings of its 2022 State of Observability and Log management Report. Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. All respondents are IT professionals responsible for managing the availability of cloud application and infrastructure environments with at least 10 TB of log data. Enterprises surveyed have at least 100 employees.

VMblog spoke with Stela Udovicic, Senior Vice President, Marketing at Era Software, to understand more. 

Stela Udovicic: For this survey, we define observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of log, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.

Organizations have harnessed insights from log data for a long time. Our research confirms that demand for log data insights is not only still strong but that those insights are critically important. Use cases are numerous and diverse across IT, security, and business stakeholders.

According to 84% of respondents, troubleshooting and monitoring applications and infrastructure environments is the leading use case for log data for IT. It is closely followed by improving security (74%) and supporting IT audits (70%). Other popular use cases include optimizing performance, capacity planning, and evaluating risk.

An overwhelming majority (96%) of IT professionals surveyed report that volumes of log data in their organizations are exploding. This growth is fueled by data coming from infrastructure (storage, network, CPU, VMs, etc.), security, cloud services, containerized applications, microservices, Kubernetes, and more. Therefore, it is interesting that log data from content delivery networks, such as Cloudflare, is seen as a major standalone source of insights.

While most respondents agree about the growth of log data volumes, there is no consensus on how much log data will grow within the next 12 months (2022). 36% of those surveyed estimated that log volumes will grow by more than 50%, and about a fifth of those surveyed think that growth will be a quintuple of 2021 growth. Now that's only within one year. Despite not having consensus on actual growth numbers projected over five years, the amount of log data will skyrocket.

VMblog: What did you find most surprising about the survey results?

Udovicic: Many things were surprising compared to other sources' study data from previous years. In this survey, respondents mention that IT teams struggle to keep up with observability data growth and take various management approaches.

76% take steps to minimize the overall growth of log data volume. The variety of methods used includes deleting log data or disabling logging when it's deemed not to be needed. This approach is risky because how do you know you'll not run into an issue when logging is disabled. The most popular method, selected by 62% of the respondents, is storing just the most critical data types.

Interestingly, IT executives are less likely to report attempts to minimize the growth of log data volumes and are more concerned with overall costs. 78% of respondents attempt to manage costs with popular cost reduction methods ranging from storing data offline to routing logs to less expensive tooling or using open-source tooling. 

However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed results. This becomes a scary proposition when, as reported, they ultimately need the erased data for troubleshooting or forensics analysis. Also, if log data is stored offline, that data is difficult to access. Therefore, I would note that it's essential to consider easy retrieval of data in the long term when managing both costs and data volumes.

VMblog: Why are organizations having difficulty with their observability and log management?

Udovicic: Before harnessing actionable insights from log and other observability data, IT needs to collect, process, store, and analyze data. According to 51% of the respondents, preparing, filtering, and cleaning data is the hardest step in that process. The second most challenging step is storing log data cost-effectively and accessible way.

Also, a frequently mentioned challenge is event correlation. Finally, building dashboards to get insights from data is reported as time consuming, too. IT teams spend time building dashboards to gain insights from their data.

According to 60% of those surveyed, analyzing data from various monitoring tools is challenging. When engineers focus on maintaining tooling, they are not innovating.

Incredibly, 97% report risks associated with scalability issues with log data tools, with 66% also reporting that if their log management tools don't scale, then troubleshooting takes longer, and 62% report delays in incident resolution. But there are also some interesting findings, such as if tools don't scale, there's more risk of accidental logging of PII data and credentials, as well as increased security risks. There is also a higher risk of false alarms and false negatives.

VMblog: What else should companies keep in mind related to data observability and log management?

Udovicic: Observability adoption is growing but still a work in progress. Only 11% of the respondents have mature observability solutions in place. Log data is the most common, has the most variety, and is the most expensive to manage compared with other observability data sources - metrics and traces.

Tracing data is also seen as very costly to manage. Because applications and microservices generate a massive amount of tracing data, organizations still struggle with the costs of managing tracing data despite various ways of sampling tracing data.

Insights from log data are critically important for organizations. It's not only IT and security teams that are reaping critical insights from log data. 83% of respondents report that business stakeholders use log data to understand customer activity.

The more mature adoption of observability within an organization drives more excitement about the growth of log data. When enterprises get valuable insights from observability and log data, such as understanding key customer and business trends, preventing outages, and reducing security risks, they see the growth of log data positively. More actionable data helps these organizations be more competitive and better meet the needs of their customers.

VMblog: With 20% of organizations reporting full deployments of observability pipelines, what does this tell you?

Udovicic: Many IT practitioners report silos of observability and log data management tools across teams. One of the methods of dealing with escalating costs of tools is using streaming pipelines (observability pipelines) to take control of costs associated with tooling.

The idea is for teams to route streams of data between the observability tools they have within their organization. For example, some data can be routed to existing log management tooling, commercial or open source, or to offline cold storage (S3, GCS, etc.).

We asked IT practitioners to describe the adoption of streaming or observability pipelines within their organization to connect, filter, process, and route log data between different tools.

We uncovered that adopting observability pipelines is a work in progress, with many organizations either having implemented or looking to implement them. And enterprise architects report higher levels of adoption when compared with IT executives or DevOps/SRE roles indicating more visibility of these roles into streaming data pipelines projects across an organization.

We expect to see continued growth in the adoption of streaming pipelines as an observability data management method to simplify costs and complexity.

VMblog: What is at stake for companies in 2022 if they don't consider a modern log management approach?

Udovicic: Log data is critical for enterprises. Both IT organizations and businesses value collecting and analyzing it. Log data volumes will continue to grow, and without innovation, IT teams will continue to struggle and apply various methods to try and manage it and associated costs.

When the adoption of modern observability tools matures, there will be more excitement around data because more value can be extracted from it.

VMblog: What steps should companies take to manage their logs better?

Udovicic: Based on this survey, here's what we recommend for organizations:

• Log data will continue to grow in 2022 and beyond, so you should invest in tools that scale easily to accommodate this growth without exponentially rising costs.
• Easy data retrieval is essential for forensics and long-term analysis, so invest in tools that allow easy historical data extraction.
• Consider not deleting data too soon as it might provide critical insights later, but instead, look for tools that can cost-efficiently manage log data long term.
• Invest in tools that integrate with existing tooling.
• For example, consider investing in streaming pipelines or observability data management tools to maximize your current investments.
• Insights from the data should be accessible across your organization - to IT, security, and business stakeholders.