Skip to content

Commit

Permalink
Update TLS bootstrapping with 1.7 features
Browse files Browse the repository at this point in the history 
This includes documenting the new CSR approver built into the
controller manager and the kubelet alpha features for certifiate
rotation.

Since the CSR approver changed over the 1.7 release cycle we need
to call out the migration steps for those using the alpha feature.
This document as a whole could probably use some updates, but the
main focus of this PR is just to get these features minimally
documented before the release.

ref:

- kubernetes/kubernetes#45030
- kubernetes/enhancements#266
- kubernetes/enhancements#267
  • Loading branch information
@ericchiang
ericchiang committed Jun 27, 2017
1 parent 0b15bf5 commit f6ed129
Showing 1 changed file with 142 additions and 40 deletions.
182 changes: 142 additions & 40 deletions docs/admin/kubelet-tls-bootstrapping.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
assignees:
- ericchiang
- mikedanese
- jcbsmpsn
title: TLS bootstrapping
---

Expand All @@ -10,10 +12,10 @@ title: TLS bootstrapping
## Overview

This document describes how to set up TLS client certificate bootstrapping for kubelets.
Kubernetes 1.4 introduces an experimental API for requesting certificates from a cluster-level
Certificate Authority (CA). The first supported use of this API is the provisioning of TLS client
Kubernetes 1.4 introduces an experimental API for requesting certificates from a cluster-level
Certificate Authority (CA). The first supported use of this API is the provisioning of TLS client
certificates for kubelets. The proposal can be found [here](https://github.com/kubernetes/kubernetes/pull/20439)
and progress on the feature is being tracked as [feature #43](https://github.com/kubernetes/features/issues/43).
and progress on the feature is being tracked as [feature #43](https://github.com/kubernetes/features/issues/43).

## apiserver configuration

Expand All @@ -22,24 +24,24 @@ This group will later be used in the controller-manager configuration to scope a
controller. As this feature matures, you should ensure tokens are bound to an RBAC policy which limits requests
using the bootstrap token to only be able to make requests related to certificate provisioning. When RBAC policy
is in place, scoping the tokens to a group will allow great flexibility (e.g. you could disable a particular
bootstrap group's access when you are done provisioning the nodes).
bootstrap group's access when you are done provisioning the nodes).

### Token auth file
Tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number
generator (such as /dev/urandom on most modern systems). There are multiple ways you can generate a token. For example:
### Token auth file
Tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number
generator (such as /dev/urandom on most modern systems). There are multiple ways you can generate a token. For example:

`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`
`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`

will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8`
will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8`

The token file will look like the following example, where the first three values can be anything and the quoted group
name should be as depicted:
The token file will look like the following example, where the first three values can be anything and the quoted group
name should be as depicted:

```
02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
```
```

Add the `--token-auth-file=FILENAME` flag to the apiserver command to enable the token file.
Add the `--token-auth-file=FILENAME` flag to the apiserver command to enable the token file.
See docs at http://kubernetes.io/docs/admin/authentication/#static-token-file for further details.

### Client certificate CA bundle
Expand All @@ -48,49 +50,149 @@ Add the `--client-ca-file=FILENAME` flag to the apiserver command to enable clie
referencing a certificate authority bundle containing the signing certificate.

## controller-manager configuration
The API for requesting certificates adds a certificate-issuing control loop to the KCM. This takes the form of a
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk.
Currently, all certificates issued have one year validity and a default set of key usages.
The API for requesting certificates adds a certificate-issuing control loop to the KCM. This takes the form of a
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk.
Currently, all certificates issued have one year validity and a default set of key usages.

### Signing assets
You must provide a Certificate Authority in order to provide the cryptographic materials necessary to issue certificates.
### Signing assets
You must provide a Certificate Authority in order to provide the cryptographic materials necessary to issue certificates.
This CA should be trusted by the apiserver for authentication with the `--client-ca-file=SOMEFILE` flag. The management
of the CA is beyond the scope of this document but it is recommended that you generate a dedicated CA for Kubernetes.
Both certificate and key are assumed to be PEM-encoded.
Both certificate and key are assumed to be PEM-encoded.

The new controller-manager flags are:
The new controller-manager flags are:
```
--cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key"
```
```

### Auto-approval
To ease deployment and testing, the alpha version of the certificate request API includes a flag to approve all certificate
requests made by users in a certain group. The intended use of this is to whitelist only the group corresponding to the bootstrap
token in the token file above. Use of this flag circumvents makes the "approval" process described below and is not recommended
for production use.
### Approval controller

In 1.7 the experimenal "group auto approver" controller is dropped in favor of a new `csrapproving` controller
that ships as part of [kube-controller-manager](/docs/admin/kube-controller-manager/) and is enabled by default.
The controller uses the [`SubjectAcessReview` API](/docs/admin/authorization/#checking-api-access) to determine
if a given user is authorized to request a CSR, then approves based on the authorization outcome. To prevent
conflicts with other approvers, the builtin approver doesn't explicitly deny CSRs, only ignoring unauthorized requests.

The controller categorizes CSRs into three subresources:

1. `nodeclient` - a request by a user for a client certificate with `O=system:nodes` and `CN=system:node:(node name)`.
2. `selfnodeclient` - a node re-requesting a client certificate with the same `O` and `CN`.
3. `selfnodeserver` - a node requesting or re-requesting a serving certificate.

The checks to determine if a CSR is a `selfndeclient` or `selfnodeserver` request is currently tied to the kubelet's
credential rotation implementation, an __alpha__ feature. As such, _the definition of `selfnodeclient` and `selfnodeserver`
will likely change in a future as the kubelet's certificate rotation graduates to beta._

The following RBAC `ClusterRoles` represent the `nodeclient`, `selfnodeclient`, and `selfnodeserver` capabilities.

```yml
# A ClusterRole which instructs the CSR approver to approve a user requesting
# node client credentials.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: approve-node-client-csr
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/nodeclient"]
verbs: ["create"]
---
# A ClusterRole which instructs the CSR approver to approve a node renewing its
# own client credentials.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: approve-node-client-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeclient"]
verbs: ["create"]
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
```
The flag is:
These powers can be granted to credentials, such as bootstrapping tokens. For example, to auto-approve all bootstrapping
tokens initial CSRs, create the following `ClusterRoleBinding`.

```yml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: auto-approve-csrs-for-group
subjects:
- kind: Group
name: system:kubelet-bootstrap
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: approve-node-client-csr
apiGroup: rbac.authorization.k8s.io
```

This `ClusterRoleBinding` is equivalent to the removed controller manager flag:

```
# REMOVED: This flag no longer works.
--insecure-experimental-approve-all-kubelet-csrs-for-group="system:kubelet-bootstrap"
```
```

## kubelet configuration
To let a node renew its own credentials, an admin would construct a `ClusterRoleBinding` such as:

```yml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: node1-client-cert-renewal
subjects:
- kind: User
name: system:node:node-1 # Let "node-1" renew its client certificate.
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: approve-node-client-renewal-csr
apiGroup: rbac.authorization.k8s.io
```

Deleting the binding would prevent the node from renewing its client credentials, and effectively
remove it from the cluster once its certificate expires.

## kubelet configuration
To use request a client cert from the certificate request API, the kubelet needs a path to a kubeconfig file that contains the
bootstrap auth token. If the file specified by `--kubeconfig` does not exist, the bootstrap kubeconfig is used to request a
client certificate from the API server. On success, a kubeconfig file referencing the generated key and obtained certificate
is written to the path specified by `--kubeconfig`. The certificate and key file will be stored in the directory pointed
by `--cert-dir`. The new flag is:
is written to the path specified by `--kubeconfig`. The certificate and key file will be stored in the directory pointed
by `--cert-dir`. The new flag is:

```
--experimental-bootstrap-kubeconfig="/path/to/bootstrap/kubeconfig"
```
--bootstrap-kubeconfig="/path/to/bootstrap/kubeconfig"
```

Additionally, in 1.7 the kubelet implements __alpha__ features for enabling rotation of both its client and/or serving certs.
These can be enabled through the respective `RotateKubeletClientCertificate` and `RotateKubeletServerCertificate` feature
flags on the kubelet, but may change in backward incompatible ways in future releases.

```
--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true
```
`RotateKubeletClientCertificate` causes the kubelet to rotate its client certificates by creating new CSRs as its existing
credentials expire. `RotateKubeletServerCertificate` causes the kubelet to both request a serving certificate after
bootstrapping its client credentials and rotate the certificate. The serving cert currently does not request DNS or IP
SANs.
## kubectl approval
The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an
## kubectl approval
The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an
"Approved" status by an appropriately-privileged user. This is intended to eventually be an automated process handled by an external
approval controller, but for the alpha version of the API it can be done manually by a cluster administrator using kubectl.
An administrator can list CSRs with `kubectl get csr`, describe one in detail with `kubectl describe <name>`. There are
[currently no direct approve/deny commands](https://github.com/kubernetes/kubernetes/issues/30163) so an approver will need to update
the Status field directly. A rough example of how to do this in bash which should only be used until the porcelain merges is available
at [https://github.com/gtank/csrctl](https://github.com/gtank/csrctl).
approval controller, but for the alpha version of the API it can be done manually by a cluster administrator using kubectl.
An administrator can list CSRs with `kubectl get csr`, describe one in detail with `kubectl describe <name>`.
Certificates can be approved or denied using the `kubectl certificate` subcommand.

0 comments on commit f6ed129

Please sign in to comment.