Vault Agent Injector Annotations are not creating : External EKS

[jgourmelen]

Describe the bug

I have 2 AWS EKS Cluster.

ClusterA : Full Vault installation (Helm)
ClusterB : ExternalAddr (to ClusterA) configuration for vault injector (Helm)

Injector is OK inside CLusterA.

But, inside ClusterB injector do nothing :
My pod is always with status (1/1) and not (2/2)

I didn't find any interesting log inside vault.
I don't know what can i do.

 NAME                                    READY   STATUS    RESTARTS   AGE
 app-68b467cb54-khfnv                    1/1     Running   0          24h
 devwebapp                               1/1     Running   0          17h
 devwebapp-with-annotations              1/1     Running   0          14m
 vault-agent-injector-554c56bb59-tl99l   1/1     Running   0          15h
 

I have used this tutorial : https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault

Environment

• Kubernetes version: 1.22
  • AWS EKS 1.22
• vault-helm version: latest

[tvoran]

Hi @jgourmelen, the next place to look is in the logs of the injector pod, vault-agent-injector-554c56bb59-tl99l. If you can share the values used to deploy the chart in ClusterB, and the annotations used in the app, that would be helpful.

[dcshiman]

same issue here, the vault injector does not have any informative logs

2022-06-03T15:16:05.135Z [INFO]  handler.auto-tls: Generated CA
2022-06-03T15:16:05.140Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2022-06-03T15:16:05.235Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2022-06-03T15:16:05.248Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:16:05.255Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:16:05.256Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:16:05.256Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:16:05.256Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:16:05.256Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:33:31.275Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:33:31.686Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:33:35.804Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:33:36.310Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:33:43.307Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2022-06-03T15:33:43.891Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...

Log level = debug
Infrastructure = AWS EKS
K8S Version = 1.22
Vault Agent Image = hashicorp/vault-k8s:0.16.1
Vault Server Image = hashicorp/vault:1.10.3

I was suspecting it is an issue with MutatingWebhookConfiguration for api version admissionregistration.k8s.io/v1 so changed Match Policy: Equivalent with no luck. Here is the results for my config
kubectl describe mutatingwebhookconfigurations hashi-vault-ha-agent-injector-cfg

Name:         hashi-vault-ha-agent-injector-cfg
Namespace:
Labels:       app.kubernetes.io/instance=hashi-vault-ha
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=vault-agent-injector
Annotations:  meta.helm.sh/release-name: hashi-vault-ha
              meta.helm.sh/release-namespace: ci-tools
API Version:  admissionregistration.k8s.io/v1
Kind:         MutatingWebhookConfiguration
Metadata:
  Creation Timestamp:  2020-11-16T04:39:41Z
  Generation:          13455
  Resource Version:    277197809
  UID:                 cc6eb59c-c18b-489d-b85b-d39a7b0d16c4
Webhooks:
  Admission Review Versions:
    v1
    v1beta1
  Client Config:
    Ca Bundle:  < hidden >
    Service:
      Name:        hashi-vault-ha-agent-injector-svc
      Namespace:   ci-tools
      Path:        /mutate
      Port:        443
  Failure Policy:  Ignore
  Match Policy:    Equivalent
  Name:            vault.hashicorp.com
  Namespace Selector:
  Object Selector:
    Match Expressions:
      Key:       app.kubernetes.io/name
      Operator:  NotIn
      Values:
        vault-agent-injector
  Reinvocation Policy:  Never
  Rules:
    API Groups:

    API Versions:
      v1
    Operations:
      CREATE
      UPDATE
    Resources:
      pods
    Scope:          *
  Side Effects:     None
  Timeout Seconds:  30
Events:             <none>

The agent seems up and reachable from pods

[tvoran]

Hi folks, another place to look for clues is the kube-apiserver logs, which should report failures to call the mutating webhook service.

@dcshiman The vault injector shouldn't typically be logging Webhooks changed. Updating certs... that frequently, so I wonder if you perhaps scaled the injector deployment outside the helm chart? If two replicas of the injector are running and not coordinating, they'll both constantly update the webhook config like that. Make sure you set the number of replicas for injector only by setting injector.replicas in the chart.

[kholisrag]

any other way rather then looking at api server log? @tvoran
I faced same issue, dunno whats wrong, and vault agent injector can't inject to a pod when annotations already there?

EKS 1.23,

values.yaml

global:
  enabled: true
  tlsDisable: false
  externalVaultAddr: "${VAULT_ADDR}"
injector:
  enabled: true
  replicas: 1
  leaderElector:
    enabled: true
  metrics:
    enabled: true
  externalVaultAddr: "${VAULT_ADDR}"
  image:
    repository: "hashicorp/vault-k8s"
    tag: "1.0.0"
    pullPolicy: IfNotPresent
  agentImage:
    repository: "hashicorp/vault"
    tag: "1.11.3"
  agentDefaults:
    cpuLimit: "100m"
    cpuRequest: "10m"
    memLimit: "100Mi"
    memRequest: "10Mi"
  authPath: "auth/kubernetes"
  nodeSelector:
    kubernetes.io/os: linux
    nodegroup-name: critical-addons-tools-ng
  tolerations:
  - key: "CriticalAddonsOnly"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
server:
  enabled: false
ui:
  enabled: false
csi:
  enabled: false

Update : fixed
we need to add additional sg rule for cluster sg, inbound 443 from vpc cidr then outbound 8080 to shared nodegroup sg
then in shared nodegroup sg we add inbound 8080 from cluster sg, this required for cluster api mutating webhook to work well

[tvoran]

@kholisrag Glad you got it working! The communication requirements for vault-k8s are outlined here as well.

[barantomasz83]

HI. I will just add one qlue for others. I had same issue, reason was not default CNI (calico) for EKS.
name = "injector.hostNetwork" value = "true" - solved the problem

[saintmalik]

HI. I will just add one qlue for others. I had same issue, reason was not default CNI (calico) for EKS. name = "injector.hostNetwork" value = "true" - solved the problem

I am not getting this, can you explain better right 👀

[barantomasz83]

HI. I will just add one qlue for others. I had same issue, reason was not default CNI (calico) for EKS. name = "injector.hostNetwork" value = "true" - solved the problem

I am not getting this, can you explain better right 👀

In my case I use Calico instead of default aws-node. Calico works diffrently then default CNI, it assignes IPs from local Calico network istead of using IPs from AWS subnets. For simplicity: when you set "hostNetwork" for iunjector it will get IP not from Calico local network but from aws subnets

vault-helm/values.yaml

Lines 328 to 330 in 50f7439

	# Should the injector pods run on the host network (useful when using
	# an alternate CNI in EKS)
	hostNetwork: false