Pepperdata announced the findings from a survey of enterprise IT
professionals to understand how companies are meeting their big data
applications needs through the use of Kubernetes. The survey was conducted in March
2021, among 800 participants from a range of industries, 72 percent of which
worked at companies with between 500 and 5000 employees.
Kubernetes is rapidly becoming the standard for cloud and on-premises
clusters. It is a complex technology, and companies are struggling to properly
and effectively implement and manage these environments. The complexity of big
data applications makes resource optimization a real challenge. Unsurprisingly,
when IT doesn't have granular visibility into big data Kubernetes performance,
optimized performance and spend are hard to achieve.
"Kubernetes is increasingly being adopted by our customers for big data
applications. As a result, we see customers experiencing performance
challenges," said Ash Munshi, CEO, Pepperdata. "This survey clearly
indicates that these problems are universal and there is a need to better
optimize these big data workloads."
Survey data collected reveals a number of insights into how businesses are
adopting Kubernetes for big data applications:
- When
asked what their goals were for adopting Kubernetes for big data
workloads, 30 percent said to "improve resource utilization for
reduced cloud costs." 23 percent want to enable their migration to
the cloud; 18 percent said to shorten deployment cycles; 15 percent wanted
to make their platforms and applications cloud-agnostic; and 14 percent
said to containerize monolithic apps.
- Porting
hundreds or thousands of apps over to Kubernetes can be challenging, and
the biggest hurdles for survey respondents included initial deployment,
followed by migration, monitoring and alerting, complexity and increased
cost, and reliability, in that order.
- The
kinds of applications and workloads respondents are running, in order of
most to least, include Spark, 30 percent; Kafka, 25 percent; Presto 23
percent; AI/deep learning workloads using PyTorch or Tensorflow at 18
percent; and "other" at five percent.
- Surprisingly,
and despite how much the media writes about the move to public cloud, this
survey found that 47 percent of respondents are using Kubernetes in private
cloud environments. On-premises use made up 35 percent, and just 18
percent of respondents said they were using Kubernetes containers in
public cloud environments.
- 45%
of Kubernetes workloads are in development and testing environments, as
users move production workloads into a new resource management framework.
30 percent are doing proof-of-concept work.
- 66
percent of respondents said 75-100 percent of their big data workloads
will be on Kubernetes by the end of 2021.
- IT
operations was the clear leader-at 80 percent-in deploying Spark and other
big data apps built on Kubernetes; Engineering followed with 11 percent;
with business unit developers at just nine percent.
View the full Pepperdata Big Data and Kubernetes report