feat(leaderelection): impl leader election for HA Deployment

[Dentrax]

Fixes #720

TODO:

• update README
• pass namespace dynamically (used BindLeaderElectionFlags)
• pass durations dynamically from cmdOptions (used LeaderElectionFlag set)
• better id handle: POD-NAME_UUID (used os.Hostname + UUID)
• add e2e test
• update Helm Chart (Pod Static 1)
  • add leaderElection config
  • add podAntiAffinity example
  • enforce replica should not greater than 1 if leaderElection is not set

Signed-off-by: Furkan furkan.turkal@trendyol.com
Co-authored-by: Emin emin.aktas@trendyol.com
Co-authored-by: Yasin yasintaha.erol@trendyol.com

[k8s-ci-robot]

Hi @Dentrax. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[damemi]

/ok-to-test
I had mostly nits on docs and logging, but the overall idea looks pretty straightforward. Thanks for the contribution @Dentrax !

[damemi]

@Dentrax forgot to mention this, could you please squash your commits down too?

[ingvagabund]

@Dentrax @eminaktas would you mind extending the e2e test suite with a test validating only a single instance is active at a time? E.g.

• create 2 namespaces (e.g. A, B)
• create e.g. 5 pods in each
• wait for 5 or more seconds, configure both deschedulers to evict pods running longer than 5 seconds, each descheduler targeting a specific namespace (includeNamespaces)
• validate only pods from a single namespace (A xor B) are evicted.

This will be a bit tricky as you will need to invoke Run from the pkg/descheduler/descheduler.go or some higher function so you are running both deschedulers from within the same binary.

[Dentrax]

@Dentrax @eminaktas would you mind extending the e2e test suite with a test validating only a single instance is active at a time?

We've tried to write a simple test by following your instructions, here is test output:

=== RUN   TestLeaderElection
    e2e_leaderelection_test.go:45: Creating testing namespace e2e-testleaderelection-a
    e2e_leaderelection_test.go:52: Creating testing namespace e2e-testleaderelection-b
    e2e_leaderelection_test.go:175: Creating deployment leaderelection for namespace e2e-testleaderelection-a
    e2e_leaderelection_test.go:175: Creating deployment leaderelection for namespace e2e-testleaderelection-b
    e2e_leaderelection_test.go:203: Pod leaderelection-76dbf88fff-kpv7d not running yet, is Pending instead
    e2e_leaderelection_test.go:203: Pod leaderelection-76dbf88fff-v98g4 not running yet, is Pending instead
    e2e_leaderelection_test.go:94: starting deschedulers
I0317 12:40:08.240823   75207 leaderelection.go:248] attempting to acquire leader lease kube-system/descheduler...
I0317 12:40:08.270859   75207 leaderelection.go:258] successfully acquired lease kube-system/descheduler
I0317 12:40:11.232282   75207 leaderelection.go:248] attempting to acquire leader lease kube-system/descheduler...
    e2e_leaderelection_test.go:135: Total evicted Pods count for e2e-testleaderelection-a: is 5 and e2e-testleaderelection-b: is 14
--- PASS: TestLeaderElection (43.24s)
PASS
ok      sigs.k8s.io/descheduler/test/e2e        44.618s

We also had to put two new policy files since we couldn't pass a PolicyConfig into DeschedulerServer struct, so we bring the following files:

test/e2e/policy_leaderelection_a.yaml
test/e2e/policy_leaderelection_b.yaml

Eventually, we're asserting that only pods from a single namespace are evicted:

singleNamespaceEvicted := (podsA == 5 && podsB != 5) || (podsA != 5 && podsB == 5)

Thanks, @ingvagabund! I think we've resolved the following above reviews. I'm not sure whether it's the best way to test leader election, but waiting your thoughts.

[a7i]

Re: RBAC, I would have expected the Role definition to include the following:

  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["create"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    resourceNames: ["descheduler"]
    verbs: ["get", "patch", "delete"]

[eminaktas]

/retest

[ingvagabund]

/approve

[ingvagabund]

Few more nits I noticed. Durations are the most important here. We need to set those wisely to reduce the traffic between the descheduler instances and the apiserver.

[ingvagabund]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Dentrax, ingvagabund

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ingvagabund]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ingvagabund]

/lgtm