Claim iss is invalid

[rgruyters]

I'm trying to deploy vault-secrets-operator in my environment, but I have some issues.

I see the following message in de vault-secrets-operator pod:

{"level":"error","ts":1620239568.733615,"msg":"Could not create API client for Vault","error":"Error making API request.\n\nURL: PUT http://vault.vault.svc.cluster.local.:8200/v1/auth/kubernetes/login\nCode: 500. Errors:\n\n* claim \"iss\" is invalid","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nmain.main\n\t/workspace/main.go:56\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

Similar message in the audit log of Vault:

{
  "time": "2021-05-05T18:21:00.261045593Z",
  "type": "request",
  "auth": {
    "token_type": "default"
  },
  "request": {
    "id": "5bb0b779-217f-af39-bfc2-d1c3da5e50d9",
    "operation": "update",
    "mount_type": "kubernetes",
    "namespace": {
      "id": "root"
    },
    "path": "auth/kubernetes/login",
    "data": {
      "jwt": "hmac-sha256:3ac6b5ce56ae139a3b8088b652683320040065d80e2245e2073dbe220673e172",
      "role": "hmac-sha256:ad9b9879a7db7d76ce1b9ed21e953dcecb53ce64fefddbc4636d2c20a900f7ed"
    },
    "remote_address": "10.42.0.168"
  }
}
{
  "time": "2021-05-05T18:21:00.261954708Z",
  "type": "response",
  "auth": {
    "token_type": "default"
  },
  "request": {
    "id": "5bb0b779-217f-af39-bfc2-d1c3da5e50d9",
    "operation": "update",
    "mount_type": "kubernetes",
    "namespace": {
      "id": "root"
    },
    "path": "auth/kubernetes/login",
    "data": {
      "jwt": "hmac-sha256:3ac6b5ce56ae139a3b8088b652683320040065d80e2245e2073dbe220673e172",
      "role": "hmac-sha256:ad9b9879a7db7d76ce1b9ed21e953dcecb53ce64fefddbc4636d2c20a900f7ed"
    },
    "remote_address": "10.42.0.168"
  },
  "response": {
    "mount_type": "kubernetes"
  },
  "error": "claim \"iss\" is invalid"
}

I'm using Helm chart version 1.14.3 with the following custom values:

    replicaCount: 1
    deploymentStrategy:
      type: Recreate
    vault:
      address: "http://vault.vault"
      authMethod: kubernetes
      kubernetesPath: auth/kubernetes
      kubernetesRole: vault-secrets-operator

Here is my policy:

path "secrets/data/k3s/*" {
  capabilities = ["read"]
}

I have tried with curl command and it works:

{
  "request_id": "6a9d35f9-6722-15e7-ff34-e8c7e5081274",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "<redacted>",
    "accessor": "<redacted>",
    "policies": [
      "default",
      "vault-secrets-operator"
    ],
    "token_policies": [
      "default",
      "vault-secrets-operator"
    ],
    "metadata": {
      "role": "vault-secrets-operator",
      "service_account_name": "vault-secrets-operator",
      "service_account_namespace": "kube-system",
      "service_account_secret_name": "vault-secrets-operator-token-sl5cx",
      "service_account_uid": "<redacted>"
    },
    "lease_duration": 7200,
    "renewable": true,
    "entity_id": "<redacted>",
    "token_type": "service",
    "orphan": true
  }
}

Any idea why?

[ricoberger]

Hi @rgruyters, if I get it right from your shared code snippets, you installed the operator into the kube-system namespace, right?

Did you follow the steps from Kubernetes Auth Method to authorize the service account to access Vault?

Which version of Vault are you using?

[rgruyters]

Hi, I have followed these steps already for Cert Manager by following these steps, Configure Kubernetes Authentication.
I have tried to use a different namespace, but the issue is the same.

I'm currently running Vault version 1.7.1.

As said, it works fine when using the curl command with the JWT token from the vault-secrets-operator ServiceAccount against the API interface from Vault.

[ricoberger]

Can you exec into the operator container and check the JWT token which is used their please:

kubectl exec -it vault-secrets-operator-9f489c44b-5ntsw -c vault-secrets-operator -- sh
cat /var/run/secrets/kubernetes.io/serviceaccount/token

It could also help if you can past the JWT token from the cat command at jwt.io and check the payload data, which should look like:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "vault-secrets-operator",
  "kubernetes.io/serviceaccount/secret.name": "vault-secrets-operator-token-82jvp",
  "kubernetes.io/serviceaccount/service-account.name": "vault-secrets-operator",
  "kubernetes.io/serviceaccount/service-account.uid": "a9802bd4-c361-4b18-9d38-f2524c391f29",
  "sub": "system:serviceaccount:vault-secrets-operator:vault-secrets-operator"
}

[rgruyters]

Here you go!

(output of cat /var/run/secrets/kubernetes.io/serviceaccount/token)

{
  "aud": [
    "https://kubernetes.default.svc.cluster.local",
    "k3s"
  ],
  "exp": 1651820324,
  "iat": 1620284324,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "vault",
    "pod": {
      "name": "vault-secrets-operator-8c849ff74-wvkts",
      "uid": "1dba8907-e1bb-4457-811b-5bc6c6553b83"
    },
    "serviceaccount": {
      "name": "vault-secrets-operator",
      "uid": "b0097d04-eb07-4cd7-90e9-706f9e5357cb"
    },
    "warnafter": 1620287931
  },
  "nbf": 1620284324,
  "sub": "system:serviceaccount:vault:vault-secrets-operator"
}

(FYI, i have deployed it now in a different namespace)

[rgruyters]

When I check the token from secrets/vault-secrets-operator-* I get this:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "vault",
  "kubernetes.io/serviceaccount/secret.name": "vault-secrets-operator-token-c7jkl",
  "kubernetes.io/serviceaccount/service-account.name": "vault-secrets-operator",
  "kubernetes.io/serviceaccount/service-account.uid": "b0097d04-eb07-4cd7-90e9-706f9e5357cb",
  "sub": "system:serviceaccount:vault:vault-secrets-operator"
}

[ricoberger]

Mh normally this should be the same token. Can you check the deployment, if the correct service account is referenced there?

What also could help, if you can provide me all the commands to reproduce the issue.

[rgruyters]

$ kubectl -n vault get deploy vault-secrets-operator -o json | jq '.spec.template.spec|.serviceAccount, .serviceAccountName'
"vault-secrets-operator"
"vault-secrets-operator"

For Vault I used the following steps:

$ kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
/ $ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config
/ $ cat <<EOF | vault policy write vault-secrets-operator -
path "secrets/data/k3s/*" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: vault-secrets-operator
/ $ vault write auth/kubernetes/role/vault-secrets-operator \
    bound_service_account_names=vault-secrets-operator \
    bound_service_account_namespaces=vault \
    policies=vault-secrets-operator \
    ttl=2h
Success! Data written to: auth/kubernetes/role/vault-secrets-operator
/ $ vault secrets enable -path=secrets kv
Success! Enabled the kv secrets engine at: secrets/
/ $

For Helm Chart:

$ helm repo add ricoberger https://ricoberger.github.io/helm-charts
"ricoberger" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "gitea-charts" chart repository
...Successfully got an update from the "ricoberger" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "k8s-at-home" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm install -n vault vault-secrets-operator ricoberger/vault-secrets-operator -f ./vault-secrets-operator.yaml
NAME: vault-secrets-operator
LAST DEPLOYED: Thu May  6 19:03:35 2021
NAMESPACE: vault
STATUS: deployed
REVISION: 1
...

Here is my custom values for the Chart:

replicaCount: 1
deploymentStrategy:
  type: Recreate
vault:
  address: "http://vault"
  authMethod: kubernetes
  kubernetesPath: auth/kubernetes
  kubernetesRole: vault-secrets-operator
  reconciliationTime: "300"

Just to be sure:

$ kubectl -n vault get secrets $(kubectl -n vault get sa vault-secrets-operator -o jsonpath="{.secrets[*]['name']}")
NAME                                 TYPE                                  DATA   AGE
vault-secrets-operator-token-s4ctz   kubernetes.io/service-account-token   3      49s
$ kubectl -n vault logs vault-secrets-operator-58cbf69757-4glq8
{"level":"info","ts":1620320684.927105,"logger":"vault","msg":"Reconciliation is enabled.","ReconciliationTime":300}
{"level":"error","ts":1620320688.399339,"msg":"Could not create API client for Vault","error":"Error making API request.\n\nURL: PUT http://vault.vault.svc.cluster.local.:8200/v1/auth/kubernetes/login\nCode: 500. Errors:\n\n* claim \"iss\" is invalid","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nmain.main\n\t/workspace/main.go:56\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

[ricoberger]

Thanks for sharing. I run your provided commands and it works 🤔

Can you try to rerun the vault commands, maybe sth. went wrong there:

vault auth enable kubernetes
vault write auth/kubernetes/config token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
vault write auth/kubernetes/role/vault-secrets-operator bound_service_account_names=vault-secrets-operator bound_service_account_namespaces=vault policies=vault-secrets-operator ttl=2h

Commands for testing

kind create cluster
kubectl create ns vault
kubens vault

helm upgrade --install vault hashicorp/vault --namespace=vault --version=0.11.0 --set server.dev.enabled=true --set injector.enabled=false --set server.image.tag="1.7.1"

kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

/ $ vault write auth/kubernetes/config \
      token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
      kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
      kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

/ $ cat <<EOF | vault policy write vault-secrets-operator -
  path "secrets/data/k3s/*" {
    capabilities = ["read"]
  }
EOF
Success! Uploaded policy: vault-secrets-operator

/ $ vault write auth/kubernetes/role/vault-secrets-operator \
      bound_service_account_names=vault-secrets-operator \
      bound_service_account_namespaces=vault \
      policies=vault-secrets-operator \
      ttl=2h
Success! Data written to: auth/kubernetes/role/vault-secrets-operator

/ $ vault secrets enable -path=secrets kv
Success! Enabled the kv secrets engine at: secrets/

/ $ vault kv put secrets/data/k3s/helloworld foo=bar
Success! Data written to: secrets/data/k3s/helloworld

cat <<EOF | helm install -n vault vault-secrets-operator ricoberger/vault-secrets-operator -f -
replicaCount: 1
deploymentStrategy:
  type: Recreate
vault:
  address: "http://vault"
  authMethod: kubernetes
  kubernetesPath: auth/kubernetes
  kubernetesRole: vault-secrets-operator
  reconciliationTime: "300"
EOF

cat <<EOF | kubectl apply -f -
apiVersion: ricoberger.de/v1alpha1
kind: VaultSecret
metadata:
  name: helloworld
spec:
  path: secrets/data/k3s/helloworld
  type: Opaque
EOF
vaultsecret.ricoberger.de/helloworld created

{"level":"info","ts":1620323440.728668,"logger":"vault","msg":"Reconciliation is enabled.","ReconciliationTime":300}
{"level":"info","ts":1620323440.7438226,"logger":"vault","msg":"Renew Vault token"}
{"level":"info","ts":1620323441.1495101,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1620323441.149729,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1620323441.150398,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0506 17:50:41.150397       1 leaderelection.go:243] attempting to acquire leader lease  vault/vaultsecretsoperator.ricoberger.de...
I0506 17:50:41.164074       1 leaderelection.go:253] successfully acquired lease vault/vaultsecretsoperator.ricoberger.de
{"level":"info","ts":1620323441.1645224,"logger":"controller-runtime.manager.controller.vaultsecret","msg":"Starting EventSource","reconciler group":"ricoberger.de","reconciler kind":"VaultSecret","source":"kind source: /, Kind="}
{"level":"info","ts":1620323441.2651837,"logger":"controller-runtime.manager.controller.vaultsecret","msg":"Starting EventSource","reconciler group":"ricoberger.de","reconciler kind":"VaultSecret","source":"kind source: /, Kind="}
{"level":"info","ts":1620323441.3658218,"logger":"controller-runtime.manager.controller.vaultsecret","msg":"Starting Controller","reconciler group":"ricoberger.de","reconciler kind":"VaultSecret"}
{"level":"info","ts":1620323441.3659487,"logger":"controller-runtime.manager.controller.vaultsecret","msg":"Starting workers","reconciler group":"ricoberger.de","reconciler kind":"VaultSecret","worker count":1}
{"level":"info","ts":1620323518.3733616,"logger":"controllers.VaultSecret","msg":"Use shared client to get secret from Vault","vaultsecret":"vault/helloworld"}
{"level":"info","ts":1620323518.3733923,"logger":"vault","msg":"Read secret secrets/data/k3s/helloworld"}
{"level":"info","ts":1620323518.3896298,"logger":"controllers.VaultSecret","msg":"Creating a new Secret","vaultsecret":"vault/helloworld","Secret.Namespace":"vault","Secret.Name":"helloworld"}
{"level":"info","ts":1620323518.4048383,"logger":"controllers.VaultSecret","msg":"Use shared client to get secret from Vault","vaultsecret":"vault/helloworld"}
{"level":"info","ts":1620323518.4048867,"logger":"vault","msg":"Read secret secrets/data/k3s/helloworld"}
{"level":"info","ts":1620323518.4138598,"logger":"controllers.VaultSecret","msg":"Updating a Secret","vaultsecret":"vault/helloworld","Secret.Namespace":"vault","Secret.Name":"helloworld"}

k get secret helloworld -o yaml

[rgruyters]

I have done the same on a local dev cluster and works fine. I have deleted my Vault service on my cluster and rebuild it, but still the same issue.

The only thing that I can think of is that my cluster is deployed via k3s and running on Raspberry PI 4. (ARM)

Because this cluster is a new k3s cluster. I have destroyed and rebuild it. Same issue. Because I was using the latest channel (v1.21.0+k3s1) of k3s I moved back to the stable channel (v1.20.6+k3s1). And now it works!

The question is, why?

[trexx]

See related issue that has helped us:
external-secrets/kubernetes-external-secrets#721

[ricoberger]

Nice finding @rgruyters and thanks for sharing @trexx.

[ricoberger]

For everyone coming across this issue. This can be fixed by setting the issuer while configuring the Kubernetes auth method.

The docs are updated for Kubernetes 1.21.x and newer (see #128).