Container Blindness Doubles Infrastructure Costs in 2021

By Nick Heudecker, Senior Director, Market Strategy & Intelligence at Cribl

As enterprises deploy container technologies, they're discovering a lack of visibility into container environments. Developers and operations teams want better insight into container performance, infrastructure costs, and security issues, but the transient nature and massive scale of container deployments makes better visibility challenging. At Cribl, we call that lack of visibility "container blindness", and believe container blindness will double the amount organizations spend on application and logging infrastructure in 2021.

Businesses are coping with uncertainty by rearchitecting for real-time adaptability and resilience. Instead of crafting monolithic applications on depreciating legacy hardware, they're building modular applications on flexible, cloud-based infrastructure. These architectures and deployment possibilities allow enterprises to compose new solutions to capture rapidly changing market conditions and application loads.

Containers are the core technology enabling this agility. Containers provide consistent, portable runtime environments for applications. According to a survey from the Cloud Native Computing Foundation (CNCF), production use of containers increased from 23% in 2016 to 84% in 2019. Nearly 20% of those survey respondents report deploying over five thousand containers in production!

The challenge for infrastructure and operations (I&O) leaders is observing what all these containers are doing. With over one hundred container management tools spread across cloud and on-premises environments, aggregating logs, metrics and traces to understand infrastructure costs is, at best, daunting. Without a unified mechanism to collect and process this stream of data, observing what's happening in your container environment is impossible.

The problem is becoming acute. According to a survey from Datadog, nearly half of containers use less than 30% of requested CPU. Similar problems occur for memory, with 45% of containers using less than 30% of requested memory. Of course, you're paying for 100% of those resources, regardless of what your containers use. The amount of money I&O leaders waste on over-provisioned container deployments is staggering. It's no surprise many I&O leaders simply give up rather than try to make sense of their container infrastructure environment.

Some I&O teams lean on existing application performance monitoring (APM) systems for insight into their container deployments. Traditional APM has its own challenges with expensive per-application or per-container pricing, driving up costs even further. These price pressures force many enterprises to only install APM on a fraction of applications, leaving application monitoring and testing to customers.

Traditional monitoring solutions also drive up costs. These products commonly charge based on daily ingestion rates. Even if infrastructure and operations leaders have a mandate to control infrastructure costs, they may not have the budget to do so with existing monitoring tools. Essentially, budgeting challenges for monitoring products are driving up costs everywhere else in the enterprise.

The short lifespans of containers also complicate visibility into these environments. Most containers exist for 24-48 hours. If your monitoring application hasn't collected the data users need before the container is deleted, the opportunity is gone. The ephemeral nature of containers, one of their key benefits, also works against teams hoping to create observable environments.

All of these factors, from environment complexity to exorbitant monitoring costs, contribute to container blindness for today's I&O leaders. Resolving this blindness requires creating an observability solution that connects dynamic infrastructure with components capable of making sense of the flood of data coming from containers.

Want to learn more about Cribl's 2021 observability predictions? Join our webinar on Tuesday, January 19th at 10am PT.

##

About the Author

Nick Heudecker is the Senior Director of Market Strategy & Intelligence at Cribl, the leading observability pipeline company. Prior to joining Cribl, he spent over seven years as an industry analyst at Gartner, covering the data and analytics market.With over twenty years of experience in software, he has led engineering and product teams across multiple successful startups in the media and advertising industries.

Evidence:

https://www.cncf.io/wp-content/uploads/2020/08/CNCF_Survey_Report.pdf

https://www.datadoghq.com/container-report/