Industry executives and experts share their predictions for 2023. Read them in this 15th annual VMblog.com series exclusive.
Observability: What companies can expect to see in 2023
By Eduardo Silva, co-founder of Calyptia
Water systems make an excellent analogy for understanding the world of data. Data lakes are enormous centralized repositories, while data streams flow freely at high speeds and high volumes.
The water analogy also extends to observability, where teams analyze an ever-growing number of logs, metrics and traces to ensure systems are working at their peak levels. Just as the water gets analyzed and tested for safety in two places (at the reservoir and in the pipelines), observability data needs to be analyzed and acted on while it is still flowing through the pipeline. Waiting for the data to arrive in the data lake is insufficient, and failure to maintain constant insight into the quality of water (and data) could lead to disaster.
As businesses in every industry become more data-driven, the amount of data generated will continue to increase. The IDC Global DataSphere, a measure of how much new data is created, captured, replicated, and consumed each year, is expected to double in size from 2022 to 2026. That’s leading to a growing demand for observability tools and some emerging best practices for implementing them. Here’s what 2023 has in store for observability professionals.
Embracing open standards and a vendor-neutral approach
As companies begin to put together their observability practice, it’s easy to put all the focus on data analysis and centralizing all of its information. Choosing a single vendor to accomplish that is tempting but also short-sighted. Being locked into a single provider can hold an enterprise’s data hostage to rising licensing and storage costs: a lethal mistake to make as companies are keeping a close eye on IT budgets in 2023.
For true (and immediate) observability, a wide variety of products and platforms need to connect to each other. A CNCF survey reported that 72% of respondents employ up to nine different logging, metrics and tracing tools, while 23% use between 10 and 15. Given that huge array of solutions, it’s critical to select something that integrates with them all. That’s why open-source tools are essential to meet today’s observability challenges. With open standards and a vendor-neutral approach, users with multiple back-ends or tools don’t have to worry that their data might be favored to a single endpoint over another.
Reducing rising cloud costs
In a recent presentation at the Monte Carlo Impact Summit, Redpoint Ventures Managing Director Tomasz Tunguz shared 9 Predictions for Data in 2023. Leading off his list is a prediction that cloud data warehouses will process 75% of workloads by 2024.
As companies across all industries become more data-driven and more reliant on the cloud, they will begin to encounter unforeseen costs – in both time and money – hidden in the cloud infrastructure. Those costs include the egress fees cloud providers charge when a company wants to move data from one place to another.
A vendor-neutral observability platform that simplifies pipelines can cut down on those excessive fees and help companies avoid cloud sticker shock. It’s expensive to route, store and analyze data in a back-end database, and those costs skyrocket as distributed, dynamic IT explodes the amount of data you need to store and process. Analyzing data at the source saves teams time typically spent configuring and maintaining those pipelines. By understanding event data more quickly and comprehensively, teams can identify what data is essential and what is not. That non-essential or duplicative data can be routed to a lower-cost back-end for long-term storage and immediate cost savings.
Avoiding the noise
It’s been more than a decade since Marc Andreessen claimed “software is eating the world,” but it’s no less true today. But as software consumes all departments within an enterprise, the atmosphere gets wildly complex very fast.
It’s becoming more important than ever to collect and route data where you need it so that engineers can focus on creating business value rather than connecting data sources to data backends. By applying business logic early (in the pipelines, to extend our water analogy), teams can identify what data is essential and begin to structure it before sending it to the back end. Along with the cost savings mentioned earlier, eliminating the complexity of configuring and maintaining your observability pipelines also significantly impacts system performance.
Keeping the water (and data) pure
To return to the increasingly relevant water-based analogy for observability in 2023, a city’s water system and a company’s data system both work best when all the components work together seamlessly. In both cases, solid infrastructure reduces user costs and frustrating downtime. And like with the water supply, the end users of data don’t want to have to think about the complexities of how everything works; instead, they want to just be able to get the data they need with confidence that the pipeline is flowing correctly and securely.
##
ABOUT THE AUTHOR
Eduardo Silva is the co-founder and CEO of Calyptia. Eduardo was previously a Principal Engineer at Treasure Data, where he was a maintainer and core developer of Fluent Bit. Eduardo also worked as a Senior Software Engineer at Oracle, where he developed Ksplice patches for enterprise and desktop Linux distributions. In addition, they have experience as a CCNA/Linux Teacher and as a Google Summer of Code Organization Admin and Mentor. Eduardo is also a Project Maintainer for Fluent Bit.
Eduardo Silva is an engineer in computer sciences from Viña del Mar, Chile. Eduardo has a CCNA from Inacap and is a certified Kubernetes Administrator from The Linux Foundation.
