Era Software released the findings of the company’s 2022 State of Observability and Log Management Report. Over 315 IT executives, cloud application architects, DevOps, and site reliability engineers (SRE) took the survey, sharing perspectives on the current state of exploding data and the struggle to gather valuable insights from the data. For this survey, Era Software defines observability as an evolution of traditional monitoring towards understanding deep insights from analyzing high volumes of logs, metrics, and trace data, collected from a wide variety of modern applications and infrastructure environments.
The survey also reveals that IT teams struggle with the massive growth of data and use various methods to manage data volumes and their associated costs, including prematurely deleting log data. However, according to 78% of the respondents, attempts to manage volumes of log data have had mixed or unwanted results, such as increased incident response times or inability to access needed data.
“We understand very well the risks IT and security organizations face with scaling log management tools. Those risks range from losing critical business insights and increased troubleshooting times to inefficient incident response. We’re pushing for evolution in observability data management and will continue to provide innovative alternatives to help customers as they adopt modern solutions to these challenges,” said Todd Persen, co-founder and CEO, Era Software.
The 2022 survey suggests that two-thirds of IT organizations require dedicated engineering resources to manage their monitoring tooling, and larger organizations with more log data are more likely to have dedicated teams for tooling management.
“Compared to similar research conducted in 2021, organizations report that observability adoption jumped by 180%. Our research indicates that as organizations mature in implementing observability, the value of critical insights from their log data is greater,” said Stela Udovicic, Senior Vice President of Marketing, Era Software.
Report findings also reveal:
- Observability data log data is critically important for organizations. 83% of respondents report that business stakeholders outside of IT use insights from log data. 68% say log data is necessary, but it’s tough to work with.
- IT continues to struggle to keep up with data volumes. 78% work to reduce volumes and costs, but they miss needed data for impact troubleshooting and security analyses. While 96% of IT professionals surveyed report that volumes of log data in their organizations are exploding.
- Existing log management tools present challenges and risks related to scalability. 97% of respondents report.
- Log data is key to observability, and innovation is needed. According to the report, 79% of respondents believe the overall cost of observability data management, including log management activities, will skyrocket in 2022 if current practices and tools don’t evolve
- Problems are beyond storing data. 96% report the need also to use the data to solve business problems.
- Engineering toil. 90% report reducing engineering toil to scale tools helps IT focus on more important work.
The company asked participants about their current use of streaming pipelines (sometimes called observability pipelines or observability data management) to connect, filter, process, and route log data between different “islands” such as Elasticsearch, Logstash, and Kibana (ELK stack), Splunk, offline cold storage (S3, Google Cloud Services, etc.). Adopting observability pipelines is a work in progress, with 65% of organizations evaluating options and 41% considering assessing options.





