This paper provides an overview of best practices for hardening your
Kubernetes clusters and how you can easily achieve them using a managed
Kubernetes offering. We've included our learnings from operating large-scale
hardened Kubernetes clusters across Oracle, working with our customers,
analysts, and based on industry best practices, such as the NSA and CISA
Kubernetes Hardening Guidance.
We cover best practices across the following
topics and how you can achieve them:
- Isolate the network access.
- Control access.
- Encrypt all data and rotate keys.
- Keep underlying software secure.
- Validate the deployed software and scan for
vulnerabilities.
- Keep calm and observe.
Isolate
the network access
Access to your cluster components must be restricted
to individuals or systems that require access to the resources. The network
design must be tailored to your requirements but include the appropriate level
of isolation of the Kubernetes cluster infrastructure, the APIs to access the
control planes, and the individual pods.
Lock down access to cluster control
planes and worker nodes
Design the Kubernetes clusters so that only
the individuals or systems that need to access the cluster control planes or
worker nodes through the network can access it. You can achieve this restriction
by implementing firewall security rules to authorize only the minimum network
traffic necessary to operate.
Cloud providers automate the provisioning of
Kubernetes clusters with fully isolated infrastructure, often fully managing
the cluster's control plane. This automation makes it easy for you to use isolated
infrastructure without networking administrators and expertise.
Restrict access to the Kubernetes API
The cluster Kubernetes API endpoint is the
main entry point to deploy applications, check state, or perform other cluster
operations. We often consider authorization policies, but restricting access to
this endpoint starts with restricting the network traffic to the endpoint. You
can configure the Kubernetes API endpoint as part of a restricted subnet in
your network.
Cloud providers usually offer the ability to
configure the Kubernetes API endpoint as part of a virtual network that peers
with your company's network or restrict it to a private subnet that can only be
accessed through a bastion host. You can also configure the Kubernetes API
endpoint on a public subnet, enabling cloud services, such as a continuous
integration-continuous delivery (CI/CD) service, to access it. When using a
public subnet, we recommend restricting access to certain known network IPs.
Most cloud providers have managed services that enable these scenarios without
the need to understand networking concepts.
Control the pods communication
Network policies allow you to control the
traffic between pods or network endpoints within your cluster. They act as
firewalls within your cluster. Application deployments can use network policies
to isolate pods from other applications and namespaces.
Configure your Kubernetes clusters with a network
policy CNI to apply network policy rules, which are otherwise ignored. Many
cloud providers support network policies CNI as part of the Kubernetes cluster that
they provision.
Control
access
Kubernetes clusters support anonymous login
by default. We recommend disabling this feature, so the cluster is configured
with strong user authentication. Start with the principle of least privilege,
and authorize access to each component, user, or program only the necessary
information and resource to accomplish the task.
Most cloud providers integrate the Kubernetes
cluster with an Identity and Access Management (IAM) service that lets you
control who has access to the cluster and integrates with Kubernetes role-based
access control (RBAC) to control what users can do in the cluster. With this
control, you can easily configure administrator user roles and apply them
across multiple clusters. As an extra layer of security, some cloud providers
let you enable multi-factor authentication (MFA) as part of their IAM service.
Encrypt
all data and rotate the keys
Keep your secrets secret
Avoid confidential data in your application
code or configuration files by using Kubernetes secrets. Kubernetes secrets are
stored in the cluster control plane etcd. The encryption key for these secrets is
still stored in plain text, so we recommend storing that in a separate secrets
store.
Most cloud providers allow you to encrypt
your encryption key, offering a second-level encryption key that can be stored
in a managed key management service. Key management services offer secret
storage and rotation in highly available and durable hardware security modules
(HSM) that meet Federal Information Processing Standards (FIPS) 140-2 Security
Level 3 security certification.
Is it paranoia to encrypt all your
data?
We don't think so. Encrypt the data of your
stateful applications at rest and in-transit. Cloud providers offer managed
block and file storage services with Kubernetes and encrypt data at rest by
default. You can also choose to encrypt this data using encryption keys that you
manage and apply cloud security services to manage and rotate these keys.
Keep
the underlying software secure
Keeping Kubernetes and the operating system
up to date enables you to stay ahead of security vulnerabilities. A good
practice includes patching your clusters at least once a quarter. We recommend
treating your cluster worker nodes as non-mutable entities that are regularly
replaced with the latest images.
Cloud providers make this process easy by
providing automatic updates and an automated cordon-drain procedure.
Validate
the deployed software and scan for vulnerabilities
Start with signed and verified images
and scan regularly
An automated continuous integration and
continuous delivery (CI/CD) pipeline to build container images, push to a
registry, and deploy to Kubernetes clusters is a good start. You can implement
checks in this process to ensure that the container images have the following
characteristics:
- They're free of known critical vulnerabilities
that can cause an accidental system failure or result in malicious activity.
- They haven't been modified since they were
published to maintain their integrity.
- They're only deployed to a Kubernetes cluster
and come from a trusted source.
Cloud providers provide container image
scanning, signing, and verification. Vulnerability checks are done against the
NIST National Vulnerability Database (NVD) and MITRE CVE databases. You can
sign images and verify that the image signatures have not changed, ensuring
that the integrity of the image has not been compromised. Finally, you can enable
the cluster admission controller to accept deployments that use only signed
images.
It's a good practice to periodically run
vulnerability scans and penetration tests on your overall infrastructure. Many
cloud providers offer the ability to monitor the containers for unusual
behavior as a cloud service.
Run the applications with the least
privilege
We recommend that you run applications in
your Kubernetes cluster with non-root user privileges on immutable filesystems,
reducing the blast radius if your container gets compromised. As an extra layer
of security, we recommend enabling SELinux or AppArmor on the cluster worker
nodes, which enables more security context settings to apply to your pods. Pod security policy (PSP) is a built-in
admission controller that allows cluster administrators to enforce that pods
are configured with a minimal set of security settings (set with the pod's
security context). PSP is being deprecated in Kubernetes 1.21. The Kubernetes
guideline says to keep PSP simple while waiting for the PSP successor next
year.
Keep
calm and observe
A key component of your security hygiene is
ensuring that you have the right mechanisms to detect the ongoing behavior and acting
to remediate any breaches. You can start with turning on audit logs that offer
visibility into the cluster operations, turning on monitoring to help you
understand the health and performance of your nodes, pods, and services, and
logging to troubleshoot failures and performance issues. For monitoring and
alerting, Prometheus with Grafana is a popular choice, but alternatives exist. Send
the standard output, standard error output of each container, and the worker
nodes logs to the logging system of your choice.
With logging and monitoring, proactive,
ongoing analysis of the information is necessary. For these processes, you can
use Security Information and Event Management (SIEM) or security posture
services offered by the cloud providers.
Conclusion
Running a Kubernetes cluster in a secure manner
requires a significant amount of work and knowledge. Using a managed Kubernetes
service helps you focus on running your applications effectively. Repetitive
tasks, such as node provisioning and upgrades involving cordon-drain, are
automated. The services that are peripheral to your Kubernetes cluster can be
fully managed, such as IAM, monitoring, logging, and registry. Managed
Kubernetes cloud services bring a real value to help you focus on what matters,
your applications.
For more information on Kubernetes security
and hardening, see the following resources:
##
To hear more
about cloud native topics, join the Cloud Native Computing Foundation and cloud native community at KubeCon+CloudNativeCon North America 2021 - October 11-15, 2021
ABOUT THE AUTHOR
Greg Verstraeten, Senior Principal Product Manager,
Oracle
Greg is a Product Manager for the managed Kubernetes service (OKE) within
Oracle Cloud Infrastructure. He focuses on delivering features that makes
Kubernetes in OCI secure and easy to use. He has 20 years of experience
building infrastructure and applications, and he has been working with
Kubernetes for 4 years. Greg is French and lives in the San Francisco Bay Area
with his family.