Kubernetes : Enable and configure encryption of secret data at rest
Before going to discuss secret data encryption, first lets see how secrets work in Kubernetes.
Secrets are used to store sensitive information, like passwords or keys.
They’re similar to ConfigMaps, except that they’re stored in an encoded format.
So here are some things to keep in mind when working with Secrets.
First of all, note that Secrets are not encrypted. They’re only encoded, meaning anyone can look up the file that you created for Secrets, or get the secret object and then decode it using different methods available.
Lets see how we can decode the secret data with an example,
- First encode the secret value and then create a secret called my-secret as below,
2. Using the etcdctl
command line, read that Secret out of etcd:
command to read secret from etcd:
output:
Check the data marked with red color in output, the secret data is stored in plain text format.
3. Now get the secret value and decode it using base64 with — decode option,
Now put all together and observe the secret data, we are clearly able to decode the secret data and which is not good for sensitive data.
Now lets see how we can do encryption of secret data.
- Check whether encryption at rest is already enabled by checking
kube-apiserver
process and see if encryption-provider-config is set in kube-apiserver pod configuration.
--encryption-provider-config is not set in the above config file.
Now lets create EncryptionConfiguration object to enable the encryption of secrets at rest.
Each resources
array item is a separate config and contains a complete configuration. The resources.resources
field is an array of Kubernetes resource names (resource
or resource.group
) that should be encrypted. The providers
array is an ordered list of the possible encryption providers.
Only one provider type may be specified per entry (identity
or aescbc
may be provided, but not both in the same item). The first provider in the list is used to encrypt resources written into the storage. When reading resources from storage, each provider that matches the stored data attempts in order to decrypt the data. If no provider can read the stored data due to a mismatch in format or secret key, an error is returned which prevents clients from accessing that resource.
To create a new Secret, perform the following steps:
- Generate a 32-byte random key and base64 encode it. If you’re on Linux or macOS, run the following command:
2. Place that value in the secret field of the EncryptionConfiguration struct. Now the EncryptionConfiguration object looks as below,
3. Set the --encryption-provider-config
flag on the kube-apiserver
to point to the location of the config file.
You will need to mount the new encryption config file to the kube-apiserver
static pod.
- Create a enc under /etc/kubernetes/ folder and save the new encryption config file to
/etc/kubernetes/enc/encryptionConfig.yaml
on the control-plane node. - Edit the manifest for the
kube-apiserver
static pod:/etc/kubernetes/manifests/kube-apiserver.yaml
similarly to this:
4. Restart your API server.
Verifying that data is encrypted:
- Create a new Secret called
my-secret2
in thedefault
namespace:
2. Using the etcdctl
command line, read that Secret out of etcd:
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
get /registry/secrets/default/my-secret2 | hexdump -C
The output is similar to the below (marked in red color),
3. Verify the stored Secret is prefixed with k8s:enc:aescbc:v1:
which indicates the aescbc
provider has encrypted the resulting data.
4. Verify the Secret is correctly decrypted when retrieved via the API:
The output should contain mykey: bXlkYXRh
, with contents of mydata
encoded,
Ensure all Secrets are encrypted:
Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
The command above reads all Secrets and then updates them to apply server side encryption.