The following webhook is completely based on Banazi Cloud Vault Secrets Webhook however, it does not support configmap or secret mutations, as well as consulTemplates like the original does.
This version was rewrite to allow AWS, GCP and VAULT secrets manager, as well as treat vault secret paths as wildcard or a directory containing multiple secrets where each secret name is the key, and a single value for it.
Another variation is allowing to get explicit secrets vs all secrets from path.
This Mutation webhook will mutate a Pod based on annotations and automatically inject secrets from various secrets managers like AWS Secret Manager, GCP Secret Manager or Hashicorp Vault using its companion tool secrets-consumer-env
Please note, this is a single secret manager setup, this tool doesn't support fetching secrets from multiple secrets managers nor it should.
This mutation webhook watch for events where a new Pod is requested via the API, if the object is a Pod and it has specific annotations, the webhook will read these annotations and convert them to env vars or volumes needed for the secrets-consumer-env tool to run.
It will create an init container with secrets-consumer-env image in it, as well as an in-memory shared volume that will also be mounted for your container.
The init container will copy the binary of secrets-consumer-env into the shared volume.
The webhook will also change your command to be prefixed by the command secrets-consumer-env
Vault can authenticate to kubernetes using a kubernetes service account
It does this by another service account called vault-reviewer with an auth-delegator permissions, which allows it to pass another service account token for authentication to the kubernetes master.
Once the authentication to kubernetes is successful, Vault returns a client token that can be used to login to Vault, Vault will check a mapping between a vault role, service account, namespace and the policy to allows/deny the access.
This vault-reviewer service account token will be configured inside the vault using vault CLI.
let’s create the service account for that vault-reviewer
Create the service account for that vault-reviewer:
Please note; if you have set up Vault on any other namespace, make sure to update this file accordingly.
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-reviewer
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-reviewer
namespace: defaultkubectl apply -f vault-reviewer.yamlEnable the Kubernetes auth backend:
Make sure you are logged in to vault using the root token
$ vault login
Token (will be hidden):$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/Configure Vault with the vault-reviewer token and Kubernetes CA:
Note: if you setup vault on any other namespace set the -n flag after each kubectl command
VAULT_SA_TOKEN_NAME=$(kubectl get sa vault-reviewer -o jsonpath="{.secrets[*]['name']}")
SA_JWT_TOKEN=$(kubectl get secret "$VAULT_SA_TOKEN_NAME" -o jsonpath="{.data.token}" | base64 --decode; echo)
SA_CA_CRT=$(kubectl get secret "$VAULT_SA_TOKEN_NAME" -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
$ vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host=https://kubernetes.default kubernetes_ca_cert="$SA_CA_CRT"
Success! Data written to: auth/kubernetes/configvault write auth/kubernetes/role/tester \
bound_service_account_names=tester \
bound_service_account_namespaces=default \
policies=test_policy \
ttl=1hvault auth enable gcpConfigure the auth method credentials:
vault write auth/gcp/config credentials=@/path/to/credentials.jsonIf you are using instance credentials or want to specify credentials via an environment variable, you can skip this step.
vault write auth/gcp/role/my-iam-role \
type="iam" \
policies="dev,prod" \
bound_service_accounts="my-service@my-project.iam.gserviceaccount.com"vault write auth/gcp/role/my-gce-role \
type="gce" \
policies="dev,prod" \
bound_projects="my-project1,my-project2" \
bound_zones="us-east1-b" \
bound_labels="foo:bar,zip:zap" \
bound_service_accounts="my-service@my-project.iam.gserviceaccount.com"For iam-type Vault roles, Vault can be given the following roles:
roles/iam.serviceAccountKeyAdminFor gce-type Vault roles, Vault can be given the following roles:
roles/compute.viewerIf you instead wish to create a custom role with only the exact GCP permissions required, use the following list of permissions:
iam.serviceAccounts.get
iam.serviceAccountKeys.get
compute.instances.get
compute.instanceGroups.listNote that the previously mentioned permissions are given to the Vault servers. The IAM service account or GCE instance that is authenticating against Vault must have the following role:
roles/iam.serviceAccountTokenCreatorBefore you install this chart you must create a namespace for it, this is due to the order in which the resources in the charts are applied (Helm collects all of the resources in a given Chart and it's dependencies, groups them by resource type, and then installs them in a predefined order (see here - Helm 2.10).
Note: this namespace is excluded from the mutations
The MutatingWebhookConfiguration gets created before the actual backend Pod which serves as the webhook itself, Kubernetes would like to mutate that pod as well, but it is not ready to mutate yet (infinite recursion in logic).
The namespace must have a label of name with the namespace name as it's value.
set the target namespace name or skip for the default name: vswh
export WEBHOOK_NS=`<namespace>`WEBHOOK_NS=${WEBHOOK_NS:-secrets-consumer-wh}
kubectl create namespace "${WEBHOOK_NS}"
kubectl label ns "${WEBHOOK_NS}" name="${WEBHOOK_NS}"Use the helm chart to install the webhook:
helm upgrade --namespace ${WEBHOOK_NS} --install secrets-consumer-webhook helm-chart --waitNOTE: --wait is necessary because of Helm timing issues, please see this issue.
When Google configure the control plane for private clusters, they automatically configure VPC peering between your Kubernetes cluster’s network in a separate Google managed project.
The auto-generated rules only open ports 10250 and 443 between masters and nodes. This means that in order to use the webhook component with a GKE private cluster, you must configure an additional firewall rule to allow your masters CIDR to access your webhook pod using the port 8443.
You can read more information on how to add firewall rules for the GKE control plane nodes in the GKE docs.
The webhook will attempt to query the metadata for the container image if no explicit command is given for secrets-consumer-env to work properly, If your container is on a private repo, you can set your docker repo credentials via the imagePullSecrets attribute of the container.
You can also specify a default secret being used by the webhook for cases where a pod has no imagePullSecrets specified. To make this work you have to set the environment variables DEFAULT_IMAGE_PULL_SECRET and DEFAULT_IMAGE_PULL_SECRET_NAMESPACE when deploying the secrets-consumer-webhook. Have a look at the values.yaml of the vault-secrets-webhook helm chart to see how this is done.
NOTE: If you EC2 nodes are having ECR instance role added the webhook can request an ECR access token through that role automatically, instead of an explicit imagePullSecret
You have the option to select which secrets you want to expose to your process, or get all secrets
To explicitly select secrets from the secret manager, add an env var to your pod using the following convention:
env:
- name: <variable name to export>
value: vault:<vault key name from secret>| Name | Description | Required | Default |
|---|---|---|---|
| "aws.secret.manager/enabled" | enable the AWS secret manager | - | false |
| "aws.secret.manager/region" | AWS secret manager region | No | us-east-1 |
| "aws.secret.manager/role-arn" | AWS IAM Role to access the secret | No | - |
| "aws.secret.manager/secret-name" | secret name | Yes | - |
| "aws.secret.manager/previous-version" | if the secret is rotated, set to "true" | No | - |
| Name | Description | Required | Default |
|---|---|---|---|
| "gcp.secret.manager/enabled" | enable the GCP secret manager | - | false |
| "gcp.secret.manager/project-id" | GCP Project ID | Yes | - |
| "gcp.secret.manager/gcp-service-account-key-secret-name" | GCP IAM service account secret name (file name must be service-account.json) |
No | Google Default Application Credentials |
| "gcp.secret.manager/secret-name" | secret name | Yes | - |
| "gcp.secret.manager/secret-version" | specify the secret version as string | No | Latest |
| Name | Description | Required | Default |
|---|---|---|---|
| "vault.secret.manager/enabled" | enable the Vault secret manager | - | false |
| "vault.secret.manager/service" | Vault cluster service address | Yes | - |
| "vault.secret.manager/tls-secret" | Vault TLS secret name | No | Latest |
| "vault.secret.manager/role" | Vault role to access the secret path | Yes | - |
| "vault.secret.manager/k8s-token-path" | alternate kubernetes service account token path | No | /var/run/secrets/kubernetes.io/serviceaccount/token |
|"vault.secret.manager/path" | Vault secret path | Yes | - |
|"vault.secret.manager/secret-version" | Vault secret version (if using v2 secret engine) | Yes | - |
|"vault.secret.manager/use-secret-names-as-keys" | treat secret path ending with / as directory where secret name is the key and a single value in each | No | - |
|"vault.secret.manager/secret-config-x" | x is a numerical number for secret alpha-numeric ordering, JSON string format | Yes | - |
vault.secret.manager/secret-config-1: '{"Path": "secrets/v2/plain/secrets/path/app", "Version": "2", "use-secret-names-as-keys": "true"}'Vault can be used with 2 backend authentications (GCP / Kubernetes)
Default authentication method, you can point it at another kubernetes backend path (multi kubernetes clusters)
| Name | Description | Required | Default |
|---|---|---|---|
| "vault.secret.manager/k8s-token-path" | alternate kubernetes service account token path | No | /var/run/secrets/kubernetes.io/serviceaccount/token |
| "vault.secret.manager/auth-path" | alternate kubernetes backend auth path | No | auth/kubernetes/login |
Use GCP service account to authenticate to Vault
| Name | Description | Required | Default |
|---|---|---|---|
| "vault.secret.manager/gcp-service-account-key-secret-name" | GCP IAM service account secret name (file name must be service-account.json) to login with gcp |
No | Latest |
| "vault.secret.manager/tls-secret" | Vault TLS secret name | No | Latest |