feat: update enc key rotation docs

doc/Contribute.md

@@ -277,12 +277,12 @@ secrets to new values and restarting all affected pods so that they
 will use these new values.
 
 Most of the process is automatic. How to trigger it is explained
-in [General Secret Rotation](secret_rotation_general.md).
+in [Secret Rotation](secret_rotation.md).
 
 Beyond this, the keys used to encrypt the Cloud Controller Database
 (CCDB) can also be rotated, however, they do not exist as general
 secrets of the KubeCF deployment. This means that the general process
 explained above __does not apply__ to them.
 
 Their custom process is explained in
-[Rotating the CCDB encryption keys](secret_rotation.md).
+[CCDB encryption key rotation](encryption_key_rotation.md).

doc/encryption_key_rotation.md

@@ -0,0 +1,86 @@
+# CCDB encryption key rotation
+
+**IMPORTANT** - Always backup the database before rotating the encryption key.
+
+The key used to encrypt the database is generated the first time kubecf is deployed.
+It is based on the Helm values:
+
+```yaml
+ccdb:
+  encryption:
+    rotation:
+      key_labels:
+      - encryption_key_0
+      current_key_label: encryption_key_0
+```
+
+For each label under `key_labels`, kubecf will generate an encryption key.
+The `current_key_label` indicates which key is currently being used.
+
+In order to rotate the CCDB encryption key, add a new label to `key_labels` (keeping the old
+labels), and mark the `current_key_label` with the newly added label. Example:
+
+```yaml
+ccdb:
+  encryption:
+    rotation:
+      key_labels:
+      - encryption_key_0
+      - encryption_key_1
+      current_key_label: encryption_key_1
+```
+
+**IMPORTANT** - key labels should be less than 240 characters long.
+
+Then, update the kubecf Helm installation. After Helm finishes its updates, trigger the
+`rotate-cc-database-key` errand:
+
+**Note** - the following command assumes the Helm installation was installed to the `kubecf`
+namespace. These values may be different depending on how kubecf was installed.
+
+```sh
+kubectl patch qjob rotate-cc-database-key \
+  --namespace kubecf \
+  --type merge \
+  --patch '{"spec":{"trigger":{"strategy":"now"}}}'
+```
+
+## Importing encryption keys
+
+When you import a CCDB database (e.g. via `mysqldump`), then the corresponding encryption
+keys must be imported as well so that the data can be decrypted by the cloud controller.
+
+If the exported data has never had its encryption key rotated, then the only thing to set is
+the top level (legacy) encryption key:
+
+```yaml
+credentials:
+  cc_db_encryption_key: "initial-encryption-key"
+```
+
+After the data has been rotated, all the key labels and values need to be set like this:
+
+```yaml
+ccdb:
+  encryption:
+    rotation:
+      key_labels:
+      - NEW_KEY
+      current_key_label: NEW_KEY
+
+credentials:
+  cc_db_encryption_key: "initial-encryption-key"
+  ccdb_key_label_new_key: "new-encryption-key"
+```
+
+The imported `key_labels` must be defined **exactly** as they were set in the exporting installation.
+As long as the actual key rotation has been performed after the last change to the
+`current_key_label`, only the current key label and value need to be configured.
+
+Their values are stored under credential keys that are made from the lowercase version of
+their key names, prefixed with `ccdb_key_label_`.  In the example above, the key
+label `NEW_KEY` has the encryption key in the credential `ccdb_key_label_new_key`.
+
+All key names must conform to this regexp: `"^[a-zA-Z]+[a-zA-Z0-9_]*[a-zA-Z0-9]+$"`.
+If it doesn't, then the CCDB must be rotated to a conforming key name **before** the
+data is exported.

doc/secret_rotation.md

@@ -1,49 +1,96 @@
-# Secret rotation
-
-## CCDB encryption key
-
-**IMPORTANT** - Always backup the database before rotating the encryption key.
-
-The key used to encrypt the database is generated the first time kubecf is deployed.
-It is based on the Helm values:
-
-```yaml
-ccdb:
-  encryption:
-    rotation:
-      key_labels:
-      - encryption_key_0
-      current_key_label: encryption_key_0
-```
-
-For each label under `key_labels`, kubecf will generate an encryption key.
-The `current_key_label` indicates which key is currently being used.
-
-In order to rotate the CCDB encryption key, add a new label to `key_labels` (keeping the old
-labels), and mark the `current_key_label` with the newly added label. Example:
-
-```yaml
-ccdb:
-  encryption:
-    rotation:
-      key_labels:
-      - encryption_key_0
-      - encryption_key_1
-      current_key_label: encryption_key_1
-```
-
-**IMPORTANT** - key labels should be less than 240 characters long.
-
-Then, update the kubecf Helm installation. After Helm finishes its updates, trigger the
-`rotate-cc-database-key` errand:
-
-**Note** - the following command assumes the Helm installation is named `kubecf` and it was
-installed to the `kubecf` namespace. These values may be different depending on how kubecf was
-installed.
-
-```sh
-kubectl patch qjob rotate-cc-database-key \
-  --namespace kubecf \
-  --type merge \
-  --patch '{"spec":{"trigger":{"strategy":"now"}}}'
-```
+# Secret Rotation
+
+Note, this document explains the general rotation of secrets.
+
+The instructions to rotate the CCDB encryption keys specifically are in
+[a separate document](encryption_key_rotation.md).
+
+The audience of this document are:
+
+  - Developers working on KubeCF.
+
+  - Operators deploying KubeCF.
+
+# Background
+
+One of the features KubeCF (or rather the cf-operator it sits on top
+of) provides is the ability to declare secrets (passwords and
+certificates) and have the system automatically generate something
+suitably random for such on deployment, and distribute the results to
+the pods using them.
+
+This removes the burden from human operators to come up with lots of
+such just to have all the internal components of KubeCF properly wired
+up for secure communication.
+
+However, even with this, operators may wish to change such secrets
+from time to time, or on a schedule. In other words, re-randomize the
+board, and limit the lifetime of any particular secret.
+
+As a note on terminology, this kind of change is called
+__rotating a secret__.
+
+This document describes how this can be done, in the context of KubeCF.
+
+# Finding secrets
+
+Retrieve the list of all secrets maintained by a KubeCF deployment via
+
+    kubectl get quarkssecret --namespace kubecf
+
+To see the information about a specific secret, for example the NATS password, use
+
+    kubectl get quarkssecret --namespace kubecf kubecf.var-nats-password --output yaml
+
+Note that each quarkssecret has a corresponding regulare k8s secret it
+controls.
+
+    kubectl get secret --namespace kubecf
+    kubectl get secret --namespace kubecf kubecf.var-nats-password --output yaml
+
+# Requesting a rotation for a specific secret
+
+We keep using `kubecf.var-nats-password` as our example secret.
+
+To rotate this secret:
+
+  1. Create a YAML file for a ConfigMap of the form:
+
+         ---				   
+         apiVersion: v1			   
+         kind: ConfigMap			   
+         metadata:			   
+           name: rotate-kubecf.var-nats-password
+           labels:			   
+             quarks.cloudfoundry.org/secret-rotation: "true"
+         data:				   
+           secrets: '["kubecf.var-nats-password"]'
+
+     Note, while the name of this ConfigMap can be technically
+     anything (allowed by k8s syntax) we recommend using a name
+     derived from the name of the secret itself, to make the
+     connection clear.
+
+     Note further that while this example rotates only a single
+     secret, the `data.secrets` key accepts an array of secret names,
+     allowing the simultaneous rotation of many secrets together.
+
+  2. Apply this ConfigMap using:
+
+         kubectl apply --namespace kubecf -f /path/to/your/yaml/file
+
+  3. The cf-operator will process this ConfigMap due to the label
+
+         quarks.cloudfoundry.org/secret-rotation: "true"
+
+     and knows that it has to invoke a rotation of the referenced
+     secrets.
+
+     The actions of the cf-operator can be followed in its log.
+
+   4. After the cf-operator has done the rotation, i.e. has not only
+      changed the secrets, but also restarted all affected pods (the
+      users of the rotated secrets), delete the trigger config map
+      again:
+
+         kubectl delete --namespace kubecf -f /path/to/your/yaml/file