Virtualization Technology News and Information
The Importance of Data Ownership Through Open Standards
By Morgan McLean, Director of Product Management at Splunk and Co-founder of OpenTelemetry 

Information is the lifeblood of any organization - and its most valuable asset. We've all come to understand that. But even with that in mind, many companies are making decisions about their data that can immediately limit its value by capturing, processing and storing it within closed systems tied to a specific vendor. In this article, we'll dive into two use cases, observability and security, and explore how open source technologies can provide increased flexibility and control over your data in each.

The phrase "software is eating the world" is over ten years old (and was true long before it entered the common lexicon). However, the rate at which firms are investing in and adopting new technologies has only accelerated over the past few years. This is visible in the incredible growth of cloud platforms and related technologies we've experienced over the past decade. As a result, we're now riding a wave of growth in observability and security tools as companies need to better understand, manage and protect their cloud services, as well as accelerate their innovation velocity. It is increasingly understood that not using high-quality observability and security tools in a business is akin to a retailer not tracking their inventory.

And yet all too often, organizations continue to capture critical runtime data using agents and SDKs that are tied to a particular vendor and lock the customer into one set of analytical capabilities (and costs). Similarly, many firms rely on different vendors to fill in different pieces of their observability and security needs, each capturing their own slice of data from production services and infrastructure that can't paint the full picture of what's actually happening.

This situation is incredibly problematic for any system your organization relies on, and particularly for observability and security tools. That's because these tools only achieve their peak usefulness when they receive well-structured data from as many sources as possible. Plus, they have enough niche functionality that many companies use multiple offerings at the same time.

Open standards are the most practical solution to this: they capture the data that all of your observability and security (and other) systems need, ensure it is structured consistently, send it to all of the correct destinations and, most importantly, allow you to retain ownership over your data supply chain to enhance or change it as you see fit.

OpenTelemetry is the standard for observability

These challenges are what motivated our (and others') investment into OpenTelemetry, which defines data structures (traces, metrics, logs and more) and metadata semantics for the signals that organizations need to capture from their applications, services and infrastructure. OpenTelemetry provides a single set of SDKs and agents that capture this information and allow it to be sent to as many destinations as you need for analysis.

OpenTelemetry is the future (and increasingly the present) of observability: it works out of the box and gives software teams freedom to control what data they ingest, what processing is done to it and where it is emitted. Here are some of the most practical benefits:

  • It's easy to set up and provides the components (SDKs, agents, etc.) you need. The language-specific components are all available in the places where developers expect to find them (Maven for Java, NPM for JavaScript, etc.).
  • OpenTelemetry includes integrations for the platforms and services that you use, and every day more of these are natively emitting data with the OpenTelemetry APIs or protocol.
  • The signals captured from your services and infrastructure (distributed traces, application metrics, system metrics, logs, etc.) all have consistent resource metadata, meaning you can determine which host a trace span was captured from and which containers are running on specific hosts.
  • You can easily produce custom data or perform custom processing on the captured data before it gets exported for processing. For example, you can use an OpenTelemetry SDK to generate a custom metric for your e-commerce site's revenue, or you could add custom annotations and dimensions to all captured traces and metrics.
  • You can send your telemetry to anywhere you want, in the native OpenTelemetry Protocol (OTLP) or any other format. This allows you to use multiple observability (or other) solutions simultaneously, and can be used to transition your team smoothly between different solutions. You can even use this to dump a stream of your telemetry for offline analysis.

OpenTelemetry is already a popular and proven technology, and is the second most active Cloud Native Computing Foundation (CNCF) project, exceeded only by Kubernetes. OpenTelemetry is a true community project. Yes, Splunk is a significant  contributor and we have been since day one, but we are certainly not alone. Every major cloud and observability vendor participates and contributes to OpenTelemetry, as do major end users. What's more, we're now seeing OpenTelemetry used in new use cases, such as a new generation of development-time tools that show production insights directly alongside code.

As an aside, I've been involved in OpenTelemetry prior to day one, and it's incredible to see how the community growth and usage have accelerated. Through work and through the community, I'm often pulled into conversations with teams who want to learn more about the project and its capabilities. As recently as six months ago, most of these calls would involve me explaining the benefits of the project and how to get started with it. Today, these sessions are more focused on suggestions for their existing OpenTelemetry deployment or about the project's roadmap because the organization is already using it.

OCSF is the standard for cybersecurity

In the same way that OpenTelemetry has created a shared standard for observability, OCSF is creating a shared standard for cybersecurity by simplifying data normalization  across tools. The Open Cybersecurity Schema Framework (OCSF) is an open source project that defines cybersecurity standards by delivering an extensible framework for developing cybersecurity schemas, along with a vendor-agnostic core security schema.

Normalizing data from multiple sources requires significant time and resources. OCSF is an open source effort designed to deliver a vendor-agnostic taxonomy that will let cybersecurity teams perform data ingestion and analysis more quickly and effectively without the need for up-front normalization.

Because OCSF is an open standard, it can work in any environment or application from any solution provider. It works with existing security standards and processes. And as more cybersecurity vendors adopt OCSF standards, security data normalization will become even easier. By not having to focus on time-consuming data normalization tasks, security teams will be able to increase their focus on their core responsibility, keeping their networks and organizations safe from cyberattacks.

Let's talk!

At Splunk, we are convinced that open standards like OpenTelemetry and OCSF are the future. We are dedicated to furthering the advancement and adoption of projects that prioritize data ownership and have built tools like Splunk Observability Cloud with native integration to OpenTelemetry with our customers in mind.

We invite you to join us, either online within each community [OpenTelemetry | OCSF], or visit with us in-person at Booth #P6 or virtually at this year's KubeCon North America 2022 in Detroit, October 24-28. Contributors and open source maintainers (including me!) from both teams will be present at our booth throughout the conference.


To hear more about cloud native topics, join the Cloud Native Computing Foundation and the cloud native community at KubeCon + CloudNativeCon North America 2022 in Detroit (and virtual) from October 24-28. 



Morgan McLean is a director of product management at Splunk focused on the Splunk Observability Cloud, along with Splunk's contributions to OpenTelemetry and the agent and ingestion unification between Splunk Observability Cloud and Splunk Enterprise. Additionally, he is the co-founder of OpenCensus and OpenTelemetry, now the second largest CNCF project behind only Kubernetes. Prior to Splunk, Morgan spent five years as a product manager at Google Cloud Platform working on DevOps and observability initiatives, along with over three years at Microsoft as a program manager designing and implementing e-commerce services. Morgan has a BASc in Engineering Physics and a BA in Economics from the University of British Columbia.

Published Monday, October 17, 2022 10:00 AM by David Marshall
Filed under: ,
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<October 2022>