By John Walsh, Community Manager, CyberArk

Kubernetes has emerged as one of the most popular - and powerful - open-source platforms for container orchestration.  As the technology has matured and been widely adopted, the security risks associated with managing and keeping secrets secure in cloud-native environments is emerging as a top priority for many organizations.

In today's modern environments, virtually any application requires some sort of secret - whether it's a database password, a service token, a certificate, or a secret to establish secure connections. Threat actors know this and are actively seeking to steal and exploit secrets as a backdoor into an organization.

This has elevated the urgency for Kubernetes developers and security teams alike to ensure that secrets are properly handled and secure. While Kubernetes currently provides a Secret object to securely store secret values, this alone does not fully mitigate the risks associated with secret credentials.

As teams work to strengthen the security practices for managing secrets, here are five big pitfalls teams need to be aware as they look to bolster overall cybersecurity and increase developer efficiency and speed:

1: Failure to Encrypt

To understand why the lack of encryption is a problem, let's look at how Secretes are created.  Kubernetes secrets can be created in one of the following ways:

●       Automatically - Existing files with sensitive data can be stored as Secret objects with the kubectl command:             

kubectl create secret my-secret --from-file=./credentials.txt

●       Manually - You can author a Secret manifest in either JSON or YAML:

apiVersion: v1

kind: Secret

metadata:

  name: my-secret

data:

  username: bXktdXNlcm5hbWUK

  password: bXktcGFzc3dvcmQK

type: Opaque

 

This object is then created based on the manifest using kubectl apply.

The problem?  As the sensitive values in the example above are base64 encoded - and thus not encrypted - it is easy to decode these values with the following command:

echo bXktdXNlcm5hbWUK | base64 --decode

my-username

 

Therefore, the secret's manifest cannot be stored beside any other manifests (Deployments, Services, and so on) in Git. You don't want to store a plain-text sensitive value in your repository!

2: etcd Is Not Secure Storage

Kubernetes uses etcd as the storage for the secrets. Etcd is a distributed key/value storage known for its great performance. However, it lacks some features that are vital for sensitive data, such as an audit log, insights into key age, and automatic key rotation.

In addition, by default, Kubernetes stores secrets in etcd unencrypted, which means that anyone with access to the etcd cluster has access to the secrets.

3: Impersonating a Kubelet with Privileged Access

The failure to secure privileged access - or root access - is a big problem for many companies outside of Kubernetes environments.  But in Kubernetes - anyone with root access to the Kubernetes nodes can access all secrets by impersonating kubelet. Kubernetes currently shares all secrets with all nodes.

Preventing privilege escalation is, therefore, essential for blocking access to secrets.

4:  Exposing Plain-Text Secrets

Kubernetes can make secrets available to pods either through exposing them as environment variables or by mounting them as files containing plain-text secrets. Environment variables can easily expose sensitive values through, for example, debug logs. Plain-text files on disk are available to everyone with access to the pod.

Given the limitations Kubernetes provides in managing and storing secrets, many companies use external secret management solution, such as AWS Secrets Manager, Azure Key Vault, or Google Secret Manager, to gain security advantages in Kubernetes. 

While this approach resolves some issues inherent in Kubernetes, other - more general - secret management challenges persist.

With a third-party solution, it is still virtually impossible to ensure that:

• Secrets are available only to those containers that actually need them. Tight integration with Kubernetes is required to map secrets to containers and to get proper insight into this.
• Secrets are not stored on disk or in environment variables.
• Applications cannot leak secrets.
• Inter-system secret transfer is always TLS-encrypted.
• Secrets are regularly, automatically rotated
  

In addition, many third-party solutions don't solve the Secret Zero problem.

5: Solving the Secret Zero Problem

Secret zero may be the biggest pitfall of all.  This is when all of your secrets are protected under another secret - a single master key or ultimate secret that grants access to everything.  

Improperly designed secrets management solutions have a master key that unlocks everything.  For example, let's say you have all your passwords stored in the cloud protected with a single password (P@$$W0rd1), and then you store the master password in a text file in the cloud for convenience.

All a hacker needs to do is find or guess this one password to access everything. This master password is secret zero. It is a lot like putting all of your eggs in one basket or locking all of your keys in drawer and then putting that key in your pocket.  The job is not done after securing your secrets, you also need to secure access to your secrets.

This large, single attack point is very attractive to bad actors. While general secret security may be improved, Secret Zero provides attackers with a single, lucrative vector within poorly designed secrets management solutions that could give the attacker access to everything.

The best solution is to avoid secret zero problem entirely by leveraging authenticators that work with the underlying container orchestration systems running your applications to automatically authenticate machines requesting secrets with strong multi-factor authentication.

To authenticate secret requests, this approach uses multiple attributes only available to trusted containers such as the IP address range assigned to valid containers, securely random UUIDs, cryptographic keys, role, name and other automatically generated / configured attributes in the orchestration system of choice.

As a result, the presence of just one of these factors presented by an application would not be reason enough to authenticate it. Multiple factors must be presented for an application to be considered bona fide. 

In this scenario, a stolen credential will have no value if the machine or attacker using it can't be authenticated. This eliminates the one powerful secret that, if compromised, can allow attackers to move latterly and rifle through a company's data.  

How to Avoid The Common Pitfalls

Fortunately, developers and security teams can take several key steps to increase the security of their Kubernetes environment.

By understanding what authentication and authorization secrets you need to control and their ideal location in your Kubernetes cluster greatly aids security. Knowing the most secure way secrets can be configured, stored, and managed for your deployed pods can simplify the overall process of secrets management and, if configured well, can help make your entire environment more secure.

Ultimately, the best way to deal with a secret is to not deal with it at all. There are myriad open source tools that teams can eliminate the common pitfalls of securing secrets in Kubernetes by providing an interface with robust secrets management capabilities for consistently and securely authenticating, controlling, and auditing non-human access across tool stacks, platforms, and cloud environments.

By working together to avoid the common mistakes that put organizations at risk, developers and security teams can unite to help the Kubernetes community continue to grow while keeping their secrets safe.

##

To learn more about cloud native technology innovation, join us at KubeCon + CloudNativeCon Europe 2021 - Virtual, which will take place from May 4-7.    

ABOUT THE AUTHOR

John Walsh Community Manager, CyberArk 

 

John has served the realm as a lord security developer, product manager, and open source community manager for more than 15 years, working on cybersecurity products such as Conjur, LDAP, Firewall, JAVA Cyptography, SSH and PrivX. He has a wife, two kids, and a small patch of land in the greater Boston area, which makes him ineligible to take the black and join the "Knight's Watch," but he's still an experienced cybersecurity professional and developer.