feat: QuarksJob to rotate all secrets #1346
Conversation
This adds a job to trigger secret rotation for all generated secrets. This is useful for administrators to rotate all secrets (generally as a precaution); attempting to document manual instructions is likely to be error-prone.
| # secret-rotation is not a BOSH release. It requires ruby & kubectl. | ||
| image: | ||
| repository: splatform/fissile-stemcell-sle | ||
| tag: SLE_15_SP1-27.4 |
There was a problem hiding this comment.
Not really happy with this; suggestions welcome.
There was a problem hiding this comment.
@mook-as create an issue for creating such a base (we'll likely have to do it on OBS)
| name: secret-rotation | ||
| namespace: {{ .Release.Namespace | quote }} | ||
| labels: | ||
| {{- list $ "secret-rotation" | include "component.labels" | nindent 4 }} |
There was a problem hiding this comment.
kubecf/chart/templates/_helpers.tpl
Line 68 in d192fa2
| # secret-rotation is not a BOSH release. It requires ruby & kubectl. | ||
| image: | ||
| repository: splatform/fissile-stemcell-sle | ||
| tag: SLE_15_SP1-27.4 |
There was a problem hiding this comment.
@mook-as create an issue for creating such a base (we'll likely have to do it on OBS)
|
Hmm, doing more testing, I seem to have issues with rotating secrets: While I'm rotating everything, I'll focus on the
This leads to database connection errors (because the |
|
This is now blocked by cloudfoundry-incubator/quarks-secret#55 — it turns out that rotating the CA certificate does not rotate the certificate that was made with that CA. This means that if we rotate everything, there's a chance the CA certificate is rotated after the certificates it signs, causing those to no longer verify with the (new) CA certificate. We also have an issue with cloudfoundry-incubator/quarks-operator#1136 where the various QuarksStatefulSets do not get updated when the secrets they depend on change; this means that we end up not being able to restart things on connection loss. Fortunately that could be worked around by adding annotations to the various things (to cause them to get updated correctly), even though there's a feature that should make the annotations unnecessary. |
Attempting to get things in a state where rotation will not break the deployment.
We need the instance groups to be updated when their dependent secrets are updated, so that the pods can pick up the new secrets. Otherwise the instance groups in flight will not match the secrets stored in Kubernetes, and can cause issues later.
Print out the names of the databases we are updating in the database-seeder job, as otherwise we have no output and it is difficult to determine if we were doing anything useful.
It requires it now and falls over if it can't contact the server.
mook-as
added
the
Status: Blocked
|
@mook-as this originated as a documentation story; |
|
@viovanov I do not believe attempting to get users to do this manually is reasonable:
|
Description
This adds a job to trigger secret rotation for all generated secrets.
This is useful for administrators to rotate all secrets (generally as a precaution); attempting to document manual instructions is likely to be error-prone.
Motivation and Context
This is in support of #703 — trying to get the administrator/operator to type in a script correctly is going to be painful; much better to automate it instead.
How Has This Been Tested?
Triggered the QuarksJob locally to see that secrets are rotated.
Generally speaking though, my experiences has tended towards this actually breaking the cluster (because the database password gets out of sync). That will be filed as a bug afterwards, because the user can already manually rotate the secrets; this just automates the process.
Types of changes
Checklist:
(new service account + role + role binding; all minimally scoped)
(That will be a follow-up)