How to create and set up Kubernetes (k8s) secrets?

A secret is stored in the ETCD. ETCD stands for "/etc directory", the place where configuration files are stored on Linux systems. It is a distributed key-value store used for storing data across a large number of nodes. A secret belongs to a namespace and can only be accessed by pods within it.

A Kubernetes secret is used for sensitive information but, how safe is it?

Indeed, the whole integrity of your Kubernetes architecture depends on it. So, let us dive in and see what it got in the guts.

Note: For the purpose of this tutorial, you can run commands on minikube, which is a lightweight Kubernetes cluster, sufficient for our use.

Also, the use case will be to secretly store ssl certs and key, therefore you can already create this files using the following commands:

openssl genrsa -out ssl.key 2048

openssl req -new -x509 -key ssl.key -out ssl.cert -days 360 -subj /CN=secret-server.example.com

How to create a Kubernetes Secret ?

First, let me point out that if you have run pods before, you have created secrets. Indeed, run a pod, type the following command and see for yourself:

kubectl run nginx --image=nginx:alpine --port=8080 --restart=Never

kubectl get secrets

See that default secret? It contains the certificate pods need to securely talk to the Kubernetes API.

To create your own secret, the following command will get you started:

kubectl create secret generic ssl-key-cert --from-file=ssl.key --from-file ssl.cert

Here the secret-creation type “generic” means the secret was created from a local file.

Note: Two other secret-creation types exist but are not covered in this article:

• Docker-registry: creates a docker cfg secret for authentication with docker registry.
• TLS: creates a TLS secret from a pair of public/private key

Now check the secret is created by running the get command again. You can get more information with :

kubectl describe secret ssl-key-cert

You do not see the secret value as it is omitted so that it is not too exposed. Run the following command to see the Base64 encoded secret:

kubectl get secret ssl-key-cert -o yaml

Don’t you think this is exposed! Yes, we will debate a bit about these security issues in the last section.

Note: there are other ways to create secret, from a yaml configuration file for instance. You can check it out in the Kubernetes' documentation.

Enable your pod to use Kubernetes secrets

Now that you have created a secret you can use it in your pod by mounting it as a volume or setting an environment variable. A yaml configuration would be:

Note: All containers referring to a secret need its own volume Mounts section.

You can create your pod with the command.

kubectl create -f <your-file>

Delete and update a Kubernetes Secret

If you update a secret consumed in a volume, it will be updated within minutes (depending on your Kubernetes sync period and cache propagation delay). You do not need to restart the pods.

And finally, you can delete a secret with the following command:

kubectl delete secret ssl-key-cert

KubeSecrets and security

We have created, used, and deleted Kubernetes secrets. Before you use them in real life let me point out some security matters you need to keep in mind:

• secrets are NOT encrypted but base64 encoded in the etcd. You can decode them with a simple echo '<your-encoded-value>' | base64 --decode;
• any user who can create a pod using a secret can also see it and expose it ;
• someone with root privileges on one node can see any secrets on the API server. This is a priority issue being addressed in the following releases.

Note: These security matters are intended in the design proposal. For instance, it is considered that “Encryption of secret data and node security are orthogonal concerns”.

So, our secrets are not that safe? Not yet. To prevent security issues, it is recommended that admins carefully:

• use IAM to apply drastic role-based access control (RBAC) politic to access the etcd ;
• encrypt theETCD at rest. It is possible since the 1.13.0 beta release with Encryption Configuration but not enabled by default! ;
• safely store secrets configuration files ;
• configure Git so that secrets are not pushed in the repo with the code.

Secure your secrets with complementary technologies

Here are some complementary technologies you could use to secure your Kubernetes secrets even more.

The first easy action with high impact you should take is to avoid pushing your secrets with your code. Tools such as Git AutoDevops allow the storing of secrets out of the Gitlab repository, the secrets are then automatically created during deployment stage.

Now let us say that the Kubernetes encryption of the etcd and RBAC are not enough for your use. You might want to switch to dedicated technologies.

Vault by HashiCorp is basically as super-safe safe for secrets that will not only encrypt them but make it harder for attackers to retrieve them. It is famously used to manage secrets in Kubernetes. Basic steps are:

• start a vault server ;
• enable the Kubernetes auth method on Vault side ;
• create a service account on the Kubernetes side and a cluster role binding for delegation of authentication requests and synchronization.

Several good articles will get you started it.

Another interesting Vault feature for upgrading security standards are dynamic secrets, a new paradigm that weekly creates and destroys new secrets for making it useless to try to break RSA in ten days (with Shore’s algorithm on your quantum computer for instance ;)).
More articles to come on these complementary technologies. Follow us on LinkedIn to keep in touch.

We have together put our hands on our first Kubernetes secrets using command lines and configuration files to access them from within a pod. We discussed what to expect from them from a security point of view and gave an overview of several leads toward upgrading security. If you liked this article you can go check our other articles on DevOps technologies such as docker.