tolerationSeconds for tolerations for controller

[zerkms]

Is your feature request related to a problem?/Why is this needed
At the moment tolerations for the controller deployment https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/aws-ebs-csi-driver/templates/controller.yaml looks like this:

      tolerations:
        - operator: Exists

which effectively allows it run everywhere. Which is fine. The problem is that with this toleration kubernetes won't reschedule the controller if a node where it runs suddenly becomes unavailable.

See https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

Kubernetes automatically adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300, unless you, or a controller, set those tolerations explicitly.

(highlight is mine)

So, you run it with default replica count 2, then you get unlucky and both nodes where those replicas run die. Now you end up having no ebs controller whatsoever and it won't fix without manual intervention.

/feature

Describe the solution you'd like in detail
I think the default tolerationSeconds: 300 (configurable) should be added as well.

Describe alternatives you've considered
There is no, it's how kubernetes works.

Additional context

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ayberk]

This sounds reasonable to me.

/lifecycle frozen
/good-first-issue

[k8s-ci-robot]

@ayberk:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

This sounds reasonable to me.

/lifecycle frozen
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[imbohyun1]

Observed the managed node group update with 'Pod Eviction Failure'. While upgrading managed node group with launch template version up, the node is tainted and all the pods should be evicted. But the node running on 'ebs-csi-controller' pod is not tainted fully because the 'ebs-csi-contoller' has a toleration. kube-scheduler keeps scheduling 'ebs-csi-controller' pod to the node which should be drained, and it prevents pods eviction until it hits max retries. Created new case #758.