By Saad Malik, CTO and
Co-founder, Spectro Cloud
2017 was a turning point
for container orchestration, as other contenders for container orchestration
winner waved the white flag and announced support for Kubernetes. Kubernetes as
next-gen infrastructure not only abstracts out compute, storage, and network
for container workloads, but also provides many features traditionally only
offered in PaaS platforms, such as automated deployment, scaling, load
balancing, logging and monitoring of container based application workloads.
This, plus Kubernetes' maintenance of desired state against a declarative model
makes the application developer and operator's lives a lot easier.
Most of the initial
Kubernetes cluster deployments were based on Virtual Machines. After all,
Kubernetes is just a bunch of software components such as API server,
controller manager, scheduler, etcd, kublet, kube-proxy, along with container
run-time, and other plugins to run on a base OS. Most enterprises already had
virtualized environments and most of the public clouds were based on virtualization
technology as well, so it was very natural to start deploying Kubernetes
clusters onto VMs. However, as more Kubernetes clusters went into production,
more people started asking: Do I really have to run Kubernetes on virtual
infrastructure? Is there a reason not to bypass virtualization and directly run
Kubernetes on bare metal machines?
The answer, as so often
is the case, is it depends - there is no one-size-fits-all answer for
this question. There are both pros and cons for deploying Kubernetes on VM
based or bare metal infrastructure:
Performance: If performance is your big concern then bare
metal probably wins the day. Virtualization will introduce a tax on the
performance side. Typically the virtualization tax will introduce 5-10% performance
overhead due to context switching and the extra hypervisor layer. Also, when
multiple VMs end up being placed on the same physical machine, virtualization
may also suffer from the noisy neighbor or oversubscription problem and further
degrade the performance.
Some workloads need
direct exclusive hardware access, e.g., AI/ML workloads might benefit from
access to GPUs, or a big data workload might need high IOPS storage access.
Although hypervisors typically provide some PCI passthrough and direct storage
access capability, it can be difficult to configure and still adds 1-2%
overhead over bare metal. Edge computing use cases have emerged that require
small form factor Kubernetes clusters on hardware platforms with limited
hardware horsepower, limited bandwidth, and limited remote management
capability.
So again, for situations
where performance or footprint is the big thing, Bare metal wins!
Cost: If your primary concern is cost then you might
prefer a bare metal solution, as virtualization sometimes adds additional cost
to infrastrutcure. On AWS, the VM and bare metal machines are priced the same,
e.g., m5.24xlarge and m5.metal, both are priced at $4.608/hr on us-east-1, both
have 96 cores, 345 ECU, 384GB memory. However. if you have on-prem private clouds
like VMware vSphere, it does come with license cost on top of bare metal
infrastructure. There can be additional management cost for virtualization
operations as well. So, if cost is your primary concern, Bare metal wins!
Operations: Virtualization adds a nice layer of abstraction
over compute, storage, and network, and all modern virtualization platforms and
public clouds also provide convenient management features. Those features,
built up over more than a decade, include features such as VM image, thin
provisioning, snapshot, rollback, virtualized SDN for network
micro-segmentation, and native persistent volume support using virtualized
attachable storage. With bare metal, on the other hand, administrators have to
deal with aged management tools and protocols such as IPMI, Redfish, and PXE
for provisioning automation, many of which are not consistent between different
vendors. Also if everything else in your environment is on virtual
infrastructure then consistency can drive general operational benefit.
Bare metal does not
provide an easy way to do snapshot and rollback and may need physical
switch-based SDN solutions and external SAN/NFS for persistent volume support
in Kubernetes. However, things are improving. Kubernetes cluster-api based
orchestration with bare metal as a provider can make provisioning automation
easier. Through some CNI and CSI plugins, Kubernetes can have some native
network and storage solutions without relying on external systems. For example,
Calico can provide Layer 3 SDN natively in Kubernetes on flat network; Storage
solutions like Portworx, OpenEBS, Rook can provide persistent volume support
using hosts' local disks clustered together via distributed file system. It's
not as comprehensive as in the fully virtualized environment, but is
improving.
Long story short, at
least for now, Virtualization wins!
Wrapping it all up, as
with so many things, there is no clear answer as to whether Kubernetes should
be deployed on virtual infrastructure or bare metal. It all depends on your
needs and your use case. What's important is to understand what those needs are
and find the right solutions that cover you across the needs you have.
##
About the Author
Saad Malik is passionate about building products in
the areas of cloud, virtualization, containers, and distributed systems. In his
fifteen years of experience, Saad has shipped multiple new products in
enterprise, service provider, and consumer technologies. He is a hardcore
Trekkie and enjoys building autonomous drone tracking software.