Kubernetes : Enable and configure encryption of secret data at rest

Before going to discuss secret data encryption, first lets see how secrets work in Kubernetes.

Secrets are used to store sensitive information, like passwords or keys.

They’re similar to ConfigMaps, except that they’re stored in an encoded format.

So here are some things to keep in mind when working with Secrets.

First of all, note that Secrets are not encrypted. They’re only encoded, meaning anyone can look up the file that you created for Secrets, or get the secret object and then decode it using different methods available.

Lets see how we can decode the secret data with an example,

1. First encode the secret value and then create a secret called my-secret as below,

2. Using the etcdctl command line, read that Secret out of etcd:

command to read secret from etcd:

command to read secret from etcd

output:

Check the data marked with red color in output, the secret data is stored in plain text format.

3. Now get the secret value and decode it using base64 with — decode option,

Now put all together and observe the secret data, we are clearly able to decode the secret data and which is not good for sensitive data.

Now lets see how we can do encryption of secret data.

1. Check whether encryption at rest is already enabled by checking kube-apiserver process and see if encryption-provider-config is set in kube-apiserver pod configuration.

--encryption-provider-config is not set in the above config file.

Now lets create EncryptionConfiguration object to enable the encryption of secrets at rest.

encryptionConfig.yaml

Each resources array item is a separate config and contains a complete configuration. The resources.resources field is an array of Kubernetes resource names (resource or resource.group) that should be encrypted. The providers array is an ordered list of the possible encryption providers.

Only one provider type may be specified per entry (identity or aescbc may be provided, but not both in the same item). The first provider in the list is used to encrypt resources written into the storage. When reading resources from storage, each provider that matches the stored data attempts in order to decrypt the data. If no provider can read the stored data due to a mismatch in format or secret key, an error is returned which prevents clients from accessing that resource.

To create a new Secret, perform the following steps:

1. Generate a 32-byte random key and base64 encode it. If you’re on Linux or macOS, run the following command:

secret field

2. Place that value in the secret field of the EncryptionConfiguration struct. Now the EncryptionConfiguration object looks as below,

encryptionConfig.yaml after replacing secret field

3. Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file.

You will need to mount the new encryption config file to the kube-apiserver static pod.

• Create a enc under /etc/kubernetes/ folder and save the new encryption config file to /etc/kubernetes/enc/encryptionConfig.yaml on the control-plane node.
• Edit the manifest for the kube-apiserver static pod: /etc/kubernetes/manifests/kube-apiserver.yaml similarly to this:

4. Restart your API server.

Verifying that data is encrypted:

1. Create a new Secret called my-secret2 in the default namespace:

my-secret2

2. Using the etcdctl command line, read that Secret out of etcd:

ETCDCTL_API=3 etcdctl \
   --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
   --cert=/etc/kubernetes/pki/etcd/server.crt \
   --key=/etc/kubernetes/pki/etcd/server.key  \
   get /registry/secrets/default/my-secret2 | hexdump -C

The output is similar to the below (marked in red color),

3. Verify the stored Secret is prefixed with k8s:enc:aescbc:v1: which indicates the aescbc provider has encrypted the resulting data.

4. Verify the Secret is correctly decrypted when retrieved via the API:

The output should contain mykey: bXlkYXRh, with contents of mydata encoded,

Ensure all Secrets are encrypted:

Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

The command above reads all Secrets and then updates them to apply server side encryption.