PSA: k8s 1.21 Vault Service Account Issuer

[dirtycajunrice]

Looks like there are a few people running into the ISS claim invalid when bumping to 1.21.

Starting k8s v1.21, Service Account Issuer Discovery feature gate is now stable and enabled by default. You have 2 methods of resolving the issue.

Secure

Configure your cluster to securely serve validation. I there are too many unique environments to give an appropriate configuration example, but banzai cloud has a nice writeup to get you started.

Insecure

If you want to continue using kubernetes-external-secrets the prior way, you need to disable issuer validation.
CLI Example

$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="$K8S_HOST" \
        kubernetes_ca_cert="$SA_CA_CRT" \
        disable_iss_validation=true

[moolen]

Hej @dirtycajunrice, thanks for reporting, i've pinned this issue!

[laukaichung]

I'm using Bank Vaults and facing the same problem. Have to disable ISS claim validation.

[CodeGlitcher]

Same problem,
Will there be a fix or do we need to migrate to the generic jwt auth method permanently?

[moolen]

Im no vault user by any means but if i understand it correctly: the fix is to securely serve the jwt validation endpoint (so that the service account jwts can be validated by vault). Either make the endpoint public readable (system:unauthenticated) or somehow tell vault to use a set of credentials to access the public key (not sure how that could work).

From the above mentioned banzai cloud writeup:

// allow unauthenticated users read acces to oidc-issuer-discovery and jwks public key
kubectl create clusterrolebinding oidc-reviewer --clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated

// then you should be able to see the oidc config
curl -k "https://my-api-server/.well-known/openid-configuration"
{
  "issuer": "https://my-api-server",
  "jwks_uri": "https://my-api-server/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}


// and the public key
curl -k "https://my-api-server/openid/v1/jwks"
{
  "keys": [
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "...",
      "alg": "RS256",
      "n": "...",
      "e": "AQAB"
    }
  ]
}

*edit*

I don't this we can fix the issue in the controller. If someone knows it please point me in the right direction.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[rdavyd]

Fixed the problem by explicitly configuring JWT issuer for the vault like this (rest of arguments are from vault documentation):

vault write auth/kubernetes/config issuer="https://kubernetes.default.svc.my.nice.cluster.local" token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Alternatively that may be set via UI.
Issuer name https://kubernetes.default.svc.my.nice.cluster.local can be looked up in kube-apiserver arguments on master.

[adelca]

It would be nice to specify from what version of Vault "disable_iss_validation=true" is supported. I can see here it was added in June 2020:
hashicorp/vault#9142
People using Vault versions older than June 2020 wont be able to use this workaround.

[samvdb]

For anyone still bumping into this issue, ensure kubernetes_host has a scheme and port defined ( eg http://vault:8200 ).

To get authentication working without enabling disable_iss_validation, manually create a service account, secret and mount it in the pod.

Example:

The service account:

---
apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
    name: secret-sa
    namespace: example

The secret:

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
   name: sa-token
   annotations:
      kubernetes.io/service-account-name: secret-sa

Modify helm chart:

values.yaml

serviceAccount:
  create: false
  name: secret-sa

filesFromSecret:
   sa-secret:
     secret: sa-token
     mountPath: /var/run/secrets/kubernetes.io/serviceaccount

This will create a fixed token in the secret sa-token which is then mounted into the pod on /var/run/secrets/kubernetes.io/serviceaccount.

Note that if you do not create your own service account secret, a fresh token will be requested from the kube-api server through the use of projected volumes. This token is NOT stored in a secret. This can be confusing when troubleshooting.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.