Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Talk about key rotation downtime and "rotationPolicy: Always" #521

Merged
merged 5 commits into from
Jul 29, 2021
+141 −47
Merged

Talk about key rotation downtime and "rotationPolicy: Always" #521

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
90c4757
usage-certificate: talk about key rotation downtime
maelvls Apr 12, 2021
4232d0c
temporary cert on ingress-gce: mention that it has been solved upstream
maelvls May 26, 2021
550fc84
key rotation: explain that the cert's spec does not renew the priv key
maelvls May 26, 2021
79e1bee
key-rotation: use descriptive link name, rephrasing
maelvls Jul 29, 2021
ea1f25b
key-rotation: re-add "reissues when a change is made to the cert"
maelvls Jul 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .spelling
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,8 @@ JoshVanL
Kyverno
hardcodes
templated
manual-rotation-private-key
issuances

# As per https://tools.ietf.org/html/rfc5280, the spelling "X.509" is the
# correct spelling. The spelling "x509" and "X509" are incorrect.
Expand Down
181 changes: 134 additions & 47 deletions content/en/docs/usage/certificate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ you will interact with cert-manager to request signed certificates.
In order to issue any certificates, you'll need to configure an
[`Issuer`](../../configuration/) resource first.


## Creating Certificate Resources

A `Certificate` resource specifies fields that are used to generate certificate
Expand All @@ -40,7 +39,7 @@ spec:
renewBefore: 360h # 15d
subject:
organizations:
- jetstack
- jetstack
# The use of the common name field has been deprecated since 2000 and is
# discouraged from being used.
commonName: example.com
Expand All @@ -54,12 +53,12 @@ spec:
- client auth
# At least one of a DNS Name, URI, or IP address is required.
dnsNames:
- example.com
- www.example.com
- example.com
- www.example.com
uris:
- spiffe://cluster.local/ns/sandbox/sa/example
- spiffe://cluster.local/ns/sandbox/sa/example
ipAddresses:
- 192.168.0.5
- 192.168.0.5
# Issuer references are always required.
issuerRef:
name: ca-issuer
Expand All @@ -79,7 +78,7 @@ The `Certificate` will be issued using the issuer named `ca-issuer` in the
`sandbox` namespace (the same namespace as the `Certificate` resource).

> Note: If you want to create an `Issuer` that can be referenced by
> `Certificate` resources in *all* namespaces, you should create a
> `Certificate` resources in _all_ namespaces, you should create a
> [`ClusterIssuer`](../../concepts/issuer/#namespaces) resource and set the
> `certificate.spec.issuerRef.kind` field to `ClusterIssuer`.

Expand All @@ -91,71 +90,159 @@ The `Certificate` will be issued using the issuer named `ca-issuer` in the
> from functioning correctly
> [`#1269`](https://github.com/jetstack/cert-manager/issues/1269).

> Note: Take care when setting the `renewBefore` field to be very close to the
> `duration` as this can lead to a renewal loop, where the `Certificate` is always
> in the renewal period. Some `Issuers` set the `notBefore` field on their
> issued X.509 certificates before the issue time to fix clock-skew issues,
> leading to the working duration of a certificate to be less than the full
> duration of the certificate. For example, Let's Encrypt sets it to be one hour
> before issue time, so the actual _working duration_ of the certificate is 89
> days, 23 hours (the _full duration_ remains 90 days).

A full list of the fields supported on the Certificate resource can be found in
the [API reference documentation](../../reference/api-docs/#cert-manager.io/v1alpha2.CertificateSpec)
the [API reference documentation](../../reference/api-docs/#cert-manager.io/v1alpha2.CertificateSpec).

## Key Usages
## X.509 key usages and extended key usages {#key-usages}

cert-manager supports requesting certificates that have a number of custom key
usages and extended key usages. Although cert-manager will attempt to honor this
request, some issuers will remove, add defaults, or otherwise completely ignore
the request and is determined on an issuer by issuer basis. The `CA` and
`SelfSigned` `Issuer` will always return certificates matching the usages you have
requested.
cert-manager supports requesting certificates that have a number of [custom key
usages](https://tools.ietf.org/html/rfc5280#section-4.2.1.3) and [extended key
usages](https://tools.ietf.org/html/rfc5280#section-4.2.1.12). Although
cert-manager will attempt to honor this request, some issuers will remove, add
defaults, or otherwise completely ignore the request.
The `CA` and `SelfSigned` `Issuer` will always return certificates matching the usages you have requested.

Unless any number of usages has been set, cert-manager will set the default
requested usages of "digital signature", "key encipherment", and "server auth".
requested usages of `digital signature`, `key encipherment`, and `server auth`.
cert-manager will not attempt to request a new certificate if the current
certificate does not match the current key usages set.
certificate does not match the current key usage set.

An exhaustive list of supported key usages can be found in the [API reference
documentation](../../reference/api-docs/#cert-manager.io/v1alpha2.KeyUsage).

## Temporary Certificates whilst Issuing
## Temporary Certificates while Issuing {#temporary-certificates-whilst-issuing}

On old GKE versions (`1.10.7-gke.1` and below), when requesting certificates
[using the ingress-shim](../ingress/) alongside the
[`ingress-gce`](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress)
ingress controller, `ingress-gce`
[required](https://github.com/kubernetes/ingress-gce/pull/388) a temporary
certificate is to be present while waiting for the issuance of a signed
certificate. Note that this issue was
[solved](https://github.com/jetstack/cert-manager/issues/606#issuecomment-424397233)
in `1.10.7-gke.2`.
maelvls marked this conversation as resolved.
Show resolved Hide resolved

To work around this, you had to add the following annotation
to your Ingress objects:

```yaml
cert-manager.io/issue-temporary-certificate": "true"
```

That made sure that a temporary self-signed certificate was present in the
`Secret`. The self-signed certificate was replaced with the signed certificate
later on.

## Rotation of the private key {#rotation-private-key}

By default, the private key won't be rotated automatically. Using the setting
`rotationPolicy: Always`, the private key Secret associated with a Certificate
object can be configured to be rotated as soon as an action triggers the
reissuance of the Certificate object (see
[Actions that will trigger a rotation of the private key](#actions-triggering-private-key-rotation) below).

With `rotationPolicy: Always`, cert-manager waits until the Certificate
object is correctly signed before overwriting the `tls.key` file in the
Secret.

When [requesting certificates using ingress-shim](../ingress/), the component
`ingress-gce`, if used, requires that a temporary certificate is present while
waiting for issuance of a signed certificate when serving. To facilitate this,
if the annotation `"cert-manager.io/issue-temporary-certificate": "true"` is
present on the certificate, a self signed temporary certificate will be present
on the `Secret` until it is overwritten once the signed certificate has been
issued.
With this setting, you can expect **no downtime** if your application can
detect changes to the mounted `tls.crt` and `tls.key` using and reload them
gracefully or automatically restart.
maelvls marked this conversation as resolved.
Show resolved Hide resolved

## Configuring private key rotation
If your application only loads the private key and signed certificate once
at start up, the new certificate won't immediately be served by your
application, and you will want to either manually restart your pod with
`kubectl rollout restart`, or automate the action by running
[wave](https://github.com/wave-k8s/wave). Wave is a Secret controller that
makes sure deployments get restarted whenever a mounted Secret changes.

When a certificate is re-issued for any reason, including because it is nearing
expiry, when a change to the spec is made or a re-issuance is manually
triggered, cert-manager supports configuring the 'private key rotation policy'
to either always re-use the existing private key (the default behavior) or to
regenerate a new private key on each issuance (the recommended behavior).
{{% alert title="Re-use of private keys" color="primary" %}}

This is configured using the `spec.privateKey.rotationPolicy` like so:
Some issuers, like the built-in [Venafi
issuer](/docs/configuration/venafi/), may disallow re-using private keys.
If this is the case, you must explicitly configure the `rotationPolicy:
Always` setting for each of your Certificate objects accordingly.

{{% /alert %}}

In the following example, the certificate has been set with
`rotationPolicy: Always`:

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: my-cert
...
spec:
secretName: my-cert-tls
dnsNames:
- example.com
privateKey:
rotationPolicy: Always
rotationPolicy: Always # 🔰 Here.
```

### Actions that will trigger a rotation of the private key {#actions-triggering-private-key-rotation}

Setting the `rotationPolicy: Always` or change the Certificate's spec won't
rotate the private key immediately. In order to get the private key secret
rotated, the certificate objects must be reissued. A certificate object is
reissued with either:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Setting the `rotationPolicy: Always` or change the Certificate's spec won't
rotate the private key immediately. In order to get the private key secret
rotated, the certificate objects must be reissued. A certificate object is
reissued with either:
Setting the `rotationPolicy: Always` or changing other fields in the Certificate spec won't
rotate the private key immediately. In order to rotate the private key, the certificate objects must be reissued. A certificate object is
reissued under the following circumstances:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not showing the diff for some reason. s/or change the/or changing the/.

And maybe it would read better as:

Changing the Certificate spec does not trigger the renewal process, so if you change the value to rotationPolicy: Always and you want the private key to be rotated immediately, you will have to manually trigger the renewal by:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also:

Changing the Certificate spec does not trigger the renewal process,

This statement in isolation is not true, and may cause confusion. The majority of fields will cause a re-issuance (e.g. changing dnsNames, commonName, usages etc). Changing the rotationPolicy (or any other fields that control how a renewal happens) won't cause a re-issuance immediately.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #521 (comment), Richard proposed to remove the line:

The certificate object is reissued when a change to the certificate object's spec is made.

I think I misunderstood Richard's suggestion back in May. I should have understood Richard's comment as:

The certificate object is reissued when a change to one of the certificate object's fields commonName, dnsNames, ipAddresses, uris, emailAddresses, subject, isCA, usages, duration or issuerRef is made.

(as per RequestMatchesSpec)

Apologies for my misunderstanding.

I re-added the line about reissuing on spec change in ea1f25b.


- when the X.509 certificate is nearing expiry,
- when a reissuance is manually triggered with the following:
```sh
kubectl cert-manager renew cert-1
```
Note that the above command requires the [kubectl
cert-manager](/docs/usage/kubectl-plugin/#renew) plugin.

{{% pageinfo color="warning" %}}

**❌** Deleting the Secret resource associated with a Certificate resource is
**not a recommended solution** for manually rotating the private key. The
recommended way to manually rotate the private key is to trigger the reissuance
of the Certificate resource with the following command (requires the [`kubectl
cert-manager`](/docs/usage/kubectl-plugin/#renew) plugin):

```sh
kubectl cert-manager renew cert-1
```

There are two supported rotation policies:
{{% /pageinfo %}}

### The `rotationPolicy` setting

The possible values for `rotationPolicy` are:

| Value | Description |
| ---------------------- | ------------------------------------------------------------- |
| `Never` (default) | cert-manager reuses the existing private key on each issuance |
| `Always` (recommended) | cert-manager regenerates a new private key on each issuance |

With `rotationPolicy: Never`, a private key is only generated if one does not
already exist in the target Secret resource (using the `tls.key` key). All
further issuances will re-use this private key. This is the default in order to
maintain compatibility with previous releases.

With `rotationPolicy: Always`, a new private key will be generated each time an
action triggers the reissuance of the certificate object (see
[actions](#actions-triggering-private-key-rotation) above). Note that if the
private key secret already exists when creating the certificate object, the
existing private key will not be used, since the rotation mechanism also
includes the initial issuance.
maelvls marked this conversation as resolved.
Show resolved Hide resolved

{{% pageinfo color="info" %}}

* **Never** (default): a private key is only generated if one does not already exist in
the target Secret resource (using the `tls.key` key). All further issuance's will re-use
this private key. This is the default in order to maintain compatibility with previous releases.
* **Always**: a new private key will be generated each time a new certificate is issued.
It is recommended you configure this `rotationPolicy` on your Certificate resources as it
is good practice to rotate private keys when a certificate is renewed.
👉 It is recommended you configure `rotationPolicy: Always` on your certificate
resources. It is considered to be a good practice to rotate private keys when a
certificate is renewed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
👉 It is recommended you configure `rotationPolicy: Always` on your certificate
resources. It is considered to be a good practice to rotate private keys when a
certificate is renewed.
👉 We recommend that you configure `rotationPolicy: Always` on your certificate
resources. It is considered to be a good practice to rotate private keys when a
certificate is renewed.

Can we link to a reference backing up this claim good practice?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot find any good reference to support this claim. By reading some discussions:

People seem to suggest that:

  1. RSA/ECDSA private keys can't (mostly) be guessed, which means there is less of a need for renewing them (unlike passwords that are way shorter),
  2. But at the same time, an RSA/ECDSA private keys may be leaked through some old backup, leading to a possible man-in-the-middle situation.

Looking around, I found that:

  • certbot renews the private key while renewing the cert by default, but there is nothing explaining why;

  • NIST's Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations does not indicate when to renew the private key; the only bit that talks about what to do with a server private key is this:

    The security of the server’s private key is critical to the security of TLS. If the server’s private key is weak or can be obtained by a third party, the third party can masquerade as the server to
    all clients.

  • NIST's Recommendation for Key Management has a nice table but no claim on when the private key should be renewweed:
    image

  • f5-bigip suggests to renew the private key (https://support.f5.com/csp/article/K14620):

    Note: When renewing an SSL certificate from a CA, F5 recommends that you generate a new certificate signing request (CSR) and private key. Although some CAs allow you to renew a certificate by using the existing CSR on file, this method is less secure as it retains the existing private key. To generate a new CSR, follow the procedure for Creating an SSL CSR and private key.

@SgtCoDFish Do you know a place where that would be written? Or should I rather remove this claim and say that "the cert-manager team suggests users renewing the private key along with the certificate"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think one big reason why you would want to, is that if you were to renew a certificate using the same private key, and then after the initial certificates expiry learn that the private key had actually been leaked, the new certificate would be valid to use with the old private key, thus the 'rotation' didn't really provide any more protection and really just extended the lifetime of the existing certificate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

James is spot on. I don't have like an authoritative source to hand for why you'd want to rotate as much as possible, but certainly I think it's best practice.

The longer a key is used, the more likely it is to be exposed. Keys should be renewed "as often as possible" for me; not only for security reasons but also operational reasons to show that it can be done, so that if it has to be done in an emergency the org has confidence that they can carry it out.

Copy link
Member Author

@maelvls maelvls Jul 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried my best to formulate a paragraph that "justifies" our point of view. Please correct me if any of this is wrong. 🔥

👉 We recommend that you configure rotationPolicy: Always on your Certificate resources.

Rotating both the certificate and the private key simultaneously prevents the risk of issuing a certificate associated with an exposed private key.

Another benefit to renewing the private key regularly is to let you be confident that the private key rotation can be done in case of an emergency.

More generally, it is a good practice to be rotating the keys as often as possible, reducing the risk associated with compromised keys.


Some Issuer types may disallow re-using private keys. If this is the case, you must explicitly
configure the `rotationPolicy` for each of your Certificates accordingly.
{{% /pageinfo %}}

## Cleaning up Secrets when Certificates are deleted

Expand Down