By Richi “RichiH” Hartmann, Director of Community at Grafana Labs
Observability is simple in theory, and can be hard to get right. The classic definition from control theory by Rudolf E. Kálmán is easy enough: “Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.” More modern interpretations such as “If you can’t ask new questions on the fly, it’s not observability” or “If you can only determine that something is broken and not why something is broken” flow from there.
Going back even further in time to Carl Friedrich Gauß, “[…] observations are […] approximations to the truth […] this can be accomplished in no other way than by a suitable combination of more observations than the number absolutely requisite for the determination of the unknown quantities”.
Tooling incurs cost, and observability is no different; there’s a balance to be struck between emitting full data and no data, just enough to be able to determine the internal states of the systems your business depends on. Data from IBM in the 1970’s pointed to 10-20% of TCO being a reasonable amount of spending. Finding this balance requires subject matter expertise in selecting what data to emit, and often in running the pipelines, databases, alerting, and visualization tools to actually use the data.
To make matters worse, most if not all last generation systems used to take an insular approach: They focused only on infrastructure, only on physical networks, only on APM, etc.
Cloud-native complexity
Cloud-native technologies make a deliberate trade-off: They expose operators to more of the underlying system complexity in return for vastly increased horizontal scalability. As most companies have noted at some point in the cloud native journey, it can be painful to be exposed to this previously hidden complexity.
On the positive side, modern, cloud-native observability tooling is built for this scale and exposed complexity. Gone are the days when data models were inherently tied to a concept of “hosts” or when it mattered where, specifically, data is coming from.
Google would have been unable to orchestrate their services without their cluster scheduler, Borg. And they would have been unable to operate their services without their observability tool, Borgmom. Kubernetes and Prometheus are the Open Source implementations of both; this is why they are the cloud-native default.
Full-stack observability
For the first time, observability tooling exists which can break down the historic silos between different data sources. Having a holistic view of the whole stack allows for much faster time to recovery.
When a front-end service is experiencing high latency, distributed tracing will help pinpoint where a user request gets stuck or delayed. Metrics across all services make it easy to detect where error rates started to spike initially. Global logs enable everything from debugging to due diligence.
And it goes beyond just this: Holistic systems enable metadata across all data types following the same basic schema, graphs and reports looking the same, clickpaths and APIs being consistent, etc.
Not only can human operators get a faster and deeper understanding of their systems when they can seamlessly move between different systems and data types like metrics, logs, and traces. It is also easier for both humans and machines to build automation on top of this understanding. High-impact engineers strive to automate themselves out of today’s work, creating a new foundation for tomorrow’s higher-impact work.
Not limited to one silo of data, but across the whole enterprise.
Shifting left and right
It goes beyond “just” operating your services, though. By integrating CI/CD more tightly with testing solutions like Grafana k6, software engineers can find problems and regressions before their code is ever deployed to production. Those same tools and data models are also being used in production.
Capacity planning has always been hard, and it has become even more difficult with cloud-native technologies. The best predictor is often past behavior of the system. Being able to look back at specific events, and correlating with e.g. visitor and sales figures, allows you to predict and right-size your deployments. The Best Current Practice in the industry is a baseline of 13 months data retention as it allows looking back at cyclic events like Black Friday.
Dynamic scaling during unexpected rush periods is also being enabled. At a time when certain usage percentages, API latencies, or similar measures are being approached, services can be set up to scale up automatically.
In the current macroeconomic environment, cost saving is front and center for a lot of businesses. Tools like OpenCost allow us to combine operational metrics with our business metrics and cost metrics. Having insight into what specific workloads incur what specific cost at what unit price has allowed us to find impressive efficiency gains without user impact. This allows users to focus on optimizing what had the most impact.
Tying all of this together are Service Level Objectives or SLOs, a core concept of SRE practices. SLO’s tap into the notion that the one thing that really matters about your services is how your users experience them. From latency, to error rates, or simple availability: once your observability stack is mature enough you can not only get a holistic view of your services as your users experience them, but also can improve their experience when needed. In a recent observability survey conducted by Grafana Labs, most companies are either using (28%) or considering using (46%) SLOs. The report also pointed to some industries leading the way: 38% of retail/e-commerce companies use SLOs.
As our environments continue to evolve we continue to search for new ways to bring clarity and simplicity to the complex environments many organizations operate in today. Showcasing the metrics from infrastructure and applications all in one place can help to identify areas where resources and costs can be optimized resulting in more efficiencies and support in breaking down the silos that inevitably exist among teams today. Setting up SLOs to help maintain the target service level are an important indicator of observability maturity and can help make your strategy more sophisticated. At Grafana Labs we believe that disparate data sources can be brought together for greater insights and interoperability in what we call our big tent philosophy. We see metrics like those from infrastructure and application as the next phase for creating better business outcomes for our customers and users.
##
To learn more about the transformative nature of cloud native applications and open source software, join us at KubeCon + CloudNativeCon Europe 2023, hosted by the Cloud Native Computing Foundation, which takes place from April 18-21.
ABOUT THE AUTHOR
Richard “RichiH” Hartmann Director of Community, Grafana Labs
Richard “RichiH” Hartmann is the Director of Community at Grafana Labs, Prometheus team member, OpenMetrics founder, OpenTelemetry member, CNCF Technical Advisory Group Observability chair, CNCF Technical Oversight Committee member, CNCF Governing Board member, and more. He also leads, organizes, or helps run various conferences from hundreds to 18,000 attendees, including KubeCon, PromCon, FOSDEM, DENOG, DebConf, and Chaos Communication Congress. In the past, he made mainframe databases work, ISP backbones run, kept the largest IRC network on Earth running, and designed and built a datacenter from scratch.
