efs csi driver doesn't work with pod having tolerations

[infa-ssaurabh]

/kind bug

What happened?
When pod doesn't have tolerations, the efs is mounted successfully as pvc. But when we have tolerations in pod(for specific reasons) the pod goes in "ContainerCreating" status with error:
driver name efs.csi.aws.com not found in the list of registered CSI drivers

What you expected to happen?
Pod should run as expected with efs mounted as pvc when tolerations are added.

How to reproduce it (as minimally and precisely as possible)?
add tolerations to pod manifest and it will go in "ContainerCreating" status.

Anything else we need to know?: NO

Environment

• Kubernetes version (use kubectl version):
  Client: GitVersion:"v1.15.0"
  Server: GitVersion:"v1.14.9-eks-c0eccc
• Driver version:
  v0.2.0

[mikegirard]

Ran into this same issue. It has to do with the daemonset that is created for the driver:

NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
aws-node       7         7         7       7            7           <none>                        561d
ebs-csi-node   7         7         7       7            7           beta.kubernetes.io/os=linux   6d3h
efs-csi-node   5         5         5       5            5           beta.kubernetes.io/os=linux   3h21m
fluentd        5         5         5       5            5           <none>                        372d
kube-proxy     7         7         7       7            7           <none>                        561d

Notice that the ebs driver started on all nodes, even though 2 of them have taints on them (used for things like jenkins only) whereas the efs DSet only hits the untainted nodes.

[mikegirard]

Workaround until DaemonSet is updated (given standard instructions used with README):

$ kubectl edit daemonset efs-csi-node -n kube-system

... scroll down to the tolerations area and modify....
    tolerations:
        - operator: Exists
        - key: CriticalAddonsOnly
          operator: Exists

and save that. You should see the daemonset update to show a full count of nodes. First key in array probably negates the second, but I wasn't sure what the original intent of that was.

[leakingtapan]

This is a bug recently fixed on EBS csi driver. Please file a patch for fixing efs driver. Thx

[wongma7]

/close

[k8s-ci-robot]

@wongma7: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mach-kernel]

If you're running a cluster with mixed Fargate and EC2 instances, edit the efs-csi-node affinity set to avoid Fargate pods (otherwise, you will get a bunch of noise):

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: eks.amazonaws.com/compute-type
              operator: NotIn
              values:
                - fargate