This repository is an implementation of the kube-apiserver encryption provider, backed by AWS KMS.
The following guide makes several assumptions:
- You have an AWS account and permission to manage KMS keys
- You have management access to a Kubernetes API server
- You have already read the Kubernetes documentation page Encrypting Secret Data at Rest
- You have already read the Kubernetes documentation page Using a KMS provider for data encryption
- The AWS KMS encryption provider will need AWS credentials configured in order to call KMS APIs. You can read more about providing credentials by reading the AWS SDK documentation on configuring your application.
First you'll need to create a KMS master key. For more details you can read the KMS documentation on creating a key. Check the KMS pricing page for up-to-date pricing information.
KEY_ID=$(aws kms create-key --query KeyMetadata.KeyId --output text)
aws kms describe-key --key-id $KEY_ID
{
"KeyMetadata": {
"Origin": "AWS_KMS",
"KeyId": "1234abcd-12ab-34cd-56ef-1234567890ab",
"Description": "",
"KeyManager": "CUSTOMER",
"Enabled": true,
"KeyUsage": "ENCRYPT_DECRYPT",
"KeyState": "Enabled",
"CreationDate": 1502910355.475,
"Arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"AWSAccountId": "111122223333"
}
}Key aliases can be used but it is not recommended. An alias can be updated to a new key, which would break how this encryption provider works. As a result all secrets encrypted before the alias update will become unreadable.
While there are numerous ways you could deploy the aws-encryption-provider plugin, the simplest way for most installations would be a static pod on the same node as each Kubernetes API server. Below is an example pod spec, and you will need to replace the image, key ARN, and region to fit your requirements.
apiVersion: v1
kind: Pod
metadata:
name: aws-encryption-provider
namespace: kube-system
spec:
containers:
- image: 111122223333.dkr.ecr.us-west-2.amazonaws.com/aws-encryption-provider:v0.0.1
name: aws-encryption-provider
command:
- /aws-encryption-provider
- --key=arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
- --region=us-west-2
- --listen=/var/run/kmsplugin/socket.sock
ports:
- containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
volumeMounts:
- mountPath: /var/run/kmsplugin
name: var-run-kmsplugin
volumes:
- name: var-run-kmsplugin
hostPath:
path: /var/run/kmsplugin
type: DirectoryOrCreateOnce you have deployed the encryption provider on all the same nodes as your API
servers, you will need to update the kube-apiserver to use the encryption
provider by setting the --encryption-provider-config flag and with the path to
your encryption configuration file. Below is an example:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
name: aws-encryption-provider
endpoint: unix:///var/run/kmsplugin/socket.sock
cachesize: 1000
timeout: 3s
- identity: {}Don't forget, you'll need to mount the directory containing the unix socket that the KMS server is listening on into the kube-apiserver.
To use encryption provider during cluster creation, you need to ensure that its running before starting kube-apiserver. For that you need to perform the following high level steps.
Note: These steps have been verified with kops but it should be similar to any other cluster bootstrapping tool.
For exact kops instructions see KOPS.md.
You need to have encryption provider running before kube-apiserver, and to do that you can
use static pods functionality. For kops, static pod manifests are available at /etc/kubernetes/manifests. You can further use kops file assets functionality to drop
the pod spec file in that directory.
As the CNI plugin is not yet available, you need to add hostNetwork: true to pod spec.
When using hostNetwork, the port 8080 used by aws-encryption-provider conflicts with
kube-apiserver which also requires the same port. To fix this, add -health-port=:8083
to args section of pod spec above. Also change the port in containerPort and livenessProbe
sections.
Use kops lifecycle hook to run a script/container that can update the kube-apiserver
manifest (available at /etc/kubernetes/manifests) to add /var/run/kmsplugin as hostMount.
Ensure master IAM role has permissions to encrypt/decrypt using the kms. You can achieve this using additionalIAMPolicies functionality of kops.
After above changes, the modified pod-spec would look like:
apiVersion: v1
kind: Pod
metadata:
name: aws-encryption-provider
namespace: kube-system
spec:
containers:
- image: 111122223333.dkr.ecr.us-west-2.amazonaws.com/aws-encryption-provider:v0.0.1
name: aws-encryption-provider
command:
- /aws-encryption-provider
- --key=arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
- --region=us-west-2
- --listen=/var/run/kmsplugin/socket.sock
- --health-port=:8083
ports:
- containerPort: 8083
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8083
volumeMounts:
- mountPath: /var/run/kmsplugin
name: var-run-kmsplugin
hostNetwork: true
volumes:
- name: var-run-kmsplugin
hostPath:
path: /var/run/kmsplugin
type: DirectoryOrCreate- First we create a secret:
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata - Then we exec into the etcd-server:
kubectl exec -it -n kube-system $(kubectl get pods -n kube-system | grep etcd-manager-main | awk '{print $1}') bash cd /opt/etcd-v3.3.10-linux-amd64/- Then check the contents of our secret in etcd store by running the following:
ETCDCTL_API=3 etcdctl \
--key /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.key \
--cert /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt \
--cacert /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt \
--endpoints "https://etcd-a.internal.${CLUSTER}:4001" get /registry/secrets/default/secret1
-- output should be something like:
0m`�He.0�cryption-provider:�1x��%�B���#JP��J���*ȝ���@\n�96�^��ۦ�~0| *�H��
`q�*�J�.P��;&~��o#�O�8m��->8L��0�C3���A7�����~���f�V�ܬ���X��_��`�H#�D��z)+�81��qW��y��`�q��}1<LF, ��N��p����i*�aC#E�߸�s������s��l�?�a
�AźR������.��8H�4�O
If you have configured your KMS master key (CMK) to have rotation enabled, AWS will update the CMK's backing encryption key every year. (You can read more about automatic key rotation at the KMS documentation page) If you are using the aws-encryption-provider with an existing master key, but want to update your cluster to use a new KMS master key, you can by roughly following the below procedure. Be sure to read the Kubernetes documentation on rotating a decryption key, as all of those steps apply to this process.
You will need to run two encryption providers on each API server using different
keys, and you must configure them to each use a different value for the name
field and each provider must listen on a different unix socket. Below is an
example encryption configuration file for all API servers prior to using the new
key.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
# provider using old key
- kms:
name: aws-encryption-provider
endpoint: unix:///var/run/kmsplugin/socket.sock
cachesize: 1000
timeout: 3s
# provider using new Key
- kms:
name: aws-encryption-provider-2
endpoint: unix:///var/run/kmsplugin/socket2.sock
cachesize: 1000
timeout: 3s
- identity: {}After all API servers have been restarted and are able to decrypt using the new key, you can switch the order of the providers with the new key at the beginning of the list and the old key below it. After all secrets have been re-encrypted with the new key, you can remove the encryption provider using the old key from the list.
Learn how to engage with the Kubernetes community on the community page.
You can reach the maintainers of this project at:
Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.