Encryption Key Rotation

[davidnuzik]

Support Encryption Key Rotation for the Rancher 2.6 Integration.

[cjellick]

Write for how this will be implemented in k3s: k3s-io/k3s#3407

[cjellick]

@dereknola this does need backported to 1.21, please open a backport issue

[dereknola]

/backport v1.21.7+rke2r1

[rancher-max]

Validated using v1.22.5-rc2+rke2r1

• secrets encryption rotation works per the design document. There is a small caveat called out in Must restart rke2-server process from initial server node after secrets-encrypt commands #2291

Full set of steps:

# Install etcdctl
# Install rke2 with multiple servers

# Create secret
$ kubectl create secret generic secret1 -n default --from-literal=mykey=mydata

# Ensure secret is encrypted. Confirm the output shows k8s:enc:aescbc:v1:aescbckey
$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret1 | hexdump -C

# Ensure status is enabled
$ sudo rke2 secrets-encrypt status

# Disable and restart on all servers. Ensure status is disabled and is showing that on all servers. Create new secret and validate that it does NOT have encrypted data
$ sudo rke2 secrets-encrypt disable
$ sudo systemctl restart rke2-server
$ sudo rke2 secrets-encrypt status
$ kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret2 | hexdump -C

# Reenable secrets encryption and restart on all servers. Ensure status is enabled on all servers
$ sudo rke2 secrets-encrypt enable
$ sudo systemctl restart rke2-server
$ sudo rke2 secrets-encrypt status

# After each step, restart on all servers. Ensure in between restarts, status shows as NOT MATCHING and after all restarts the status is matching on all servers
$ sudo rke2 secrets-encrypt prepare
$ sudo systemctl restart rke2-server
$ sudo rke2 secrets-encrypt status
$ sudo rke2 secrets-encrypt rotate
$ sudo systemctl restart rke2-server
$ sudo rke2 secrets-encrypt status
$ sudo rke2 secrets-encrypt reencrypt
$ sudo watch rke2 secrets-encrypt status
$ sudo systemctl restart rke2-server
$ sudo rke2 secrets-encrypt status

# Create a third secret. Ensure all 3 created secrets now are using this new encryption. Output for all 3 should show something that includes the new encryption value, for example: k8s:enc:aescbc:v1:aescbckey-2021-12-16T19:38:27Z:
$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret1 | hexdump -C
$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret2 | hexdump -C
$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret3 | hexdump -C