Kubelet Evented PLEG for Better Performance

[harche]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubelet Evented PLEG for Better Performance
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3386-kubelet-evented-pleg/README.md
• Discussion Link: https://kubernetes.slack.com/archives/C03HA69L6R1 ,Add runtime events to CRI kubernetes#110165
• Primary contact (assignee): @harche, @haircommander, @ruiwen-zhao
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.25
  • Beta1 release target (x.y): 1.27 (default false)
  • Revert to alpha: 1.30 (v1.27.9、v1.28.6、v1.29.1)
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • API Changes - Update CRI API to support Evented PLEG kubernetes#111642
  • KEP (k/enhancements) update PR(s): KEP-3386: Kubelet Evented PLEG for Better Performance #3387
  • Code (k/k) update PR(s): Add Support for Evented PLEG kubernetes#111384
• Beta (default disabled)
  • KEP (k/enhancements) update
    • PR(s): KEP-3386: Graduate Evented PLEG to Beta #3817
  • Code (k/k) update PR(s):
    • Graduate Evented PLEG to Beta kubernetes#115967
    • OCPNODE-1336: Add a CI Job to run node e2e tests by enabling the evented pleg feature test-infra#28366
    • Add a periodic CI job to test the Evented PLEG feature test-infra#28592
  • Docs (k/website) updates(s):
    • Graduate Evented PLEG to Beta website#39913
• Revert to alpha
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
    • v1.30(backported): Revert "Graduate Evented PLEG to Beta" kubernetes#122697
    • v1.29: Fix issue in enabling evented pleg feature gate kubernetes#120942
    • v1.30: fix nil pointer dereference when EventedPLEG is enabled kubernetes#122475
  • Docs (k/website) update(s):
• Beta
  • Issues to fix before re-promoting it to beta:
    • Pod lifecycle broken with kubelet's EventedPLEG feature kubernetes#121003
    • blocker bugfix [Failing Test] ci-kubernetes-e2e-kind-beta-features/pull-kubernetes-e2e-kind-beta-features failed for enabling some beta features(by default disabled) kubernetes#121349 WIP in : Fix evented pleg mirror pod & use IsEventedPLEGInUse instead of FG status check kubernetes#122778
    • performance blocker issue fix EventedPLEG: Pass event created timestamp correctly to cache kubernetes#124297 @hshiina
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[harche]

/sig-node

[Priyankasaggu11929]

Hello @harche 👋, 1.25 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PST on Thursday June 23, 2022.

For note, This enhancement is targeting for stage alpha for 1.25 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

Looks like for this one, we would need to update the following:

• update the kep.yaml file in the merged KEP to add latest-milestone: "v1.25”

For note, the status of this enhancement is marked as at risk. Thank you for keeping the issue description up-to-date!

[haircommander]

I opened #3410 to target at 1.25

[Priyankasaggu11929]

/sig node

[Priyankasaggu11929]

With KEP PR #3410 merged, the enhancement is ready for the 1.25 Enhancements Freeze.

For note, the status is now marked as tracked. Thank you so much! 🙂

[Priyankasaggu11929]

Hello @haircommander @harche 👋

Checking in once more as we approach 1.25 code freeze at 01:00 UTC on Wednesday, 3rd August 2022.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.
  • Add Support for Evented PLEG kubernetes#111384

I was able to find the following draft k/k PRs for the KEP:

• Cri evented pleg approach 1 kubernetes#110911
• Cri evented pleg approach 2 kubernetes#110912

Please verify, if there are any additional k/k PRs besides the ones listed above.

Please plan to get the open/draft k/k merged by the code freeze deadline. The status of the enhancement is currently marked as at-risk.

Thank you for keeping the issue description up-to-date!

[harche]

Thanks @Priyankasaggu11929 for reaching out. I am planning to raise a PR by coming Monday.

[harche]

@Priyankasaggu11929 I just raised a PR - kubernetes/kubernetes#111384

[Priyankasaggu11929]

Hello @harche 👋

Just a gentle reminder from the enhancement team as we approach 1.25 code freeze at 01:00 UTC on Wednesday, 3rd August 2022 (which is almost 2 days from now)

Please plan to have the open k/k PR merged before then.

The status of this enhancement is currently marked as at risk

Thank you

[harche]

API Changes PR - kubernetes/kubernetes#111642

[Priyankasaggu11929]

Hello 👋, 1.25 Enhancements Lead here.

Unfortunately, this enhancement did not meet the code freeze criteria because there are still unmerged k/k code PRs.

If you still wish to progress this enhancement in v1.25, please file an exception request. Thank you so much!

/milestone clear

[harche]

@Priyankasaggu11929 Request for the exception has been raised - https://groups.google.com/g/kubernetes-sig-node/c/H_jmCppaurU/m/vwOzc-EFAAAJ

[marosset]

/milestone v1.26
/label lead-opted-in
(I'm doing this on behalf of @ruiwen-zhao / SIG-node)

[pacoxu]

Recent issues around this KEP:

• [Failing Test] ci-kubernetes-e2e-kind-beta-features/pull-kubernetes-e2e-kind-beta-features failed for enabling some beta features(by default disabled) kubernetes#121349: when the FG Evented PLEG is enabled, ci failed
  • Pod lifecycle broken with kubelet's EventedPLEG feature kubernetes#121003 This makes the pod have multiple containers running(which should be one for static pods like kube-controller-manager and kube-scheduler).
• Evented PLEG job is failing  kubernetes#120140
• Possible data race in code kubernetes#120934
• [Flaking] [sig-node] MirrorPod when create a mirror pod without changes should successfully recreate when file is removed and recreated [NodeConformance] kubernetes#122132

[pacoxu]

I opened kubernetes/kubernetes#122124 to fix the known issue mentioned v1.29.

• during local test with v1.29.0, I find another NPE and the fix PR is open in fix nil pointer dereference when EventedPLEG is enabled kubernetes#122475.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[hpx7]

Should this be reopened?

[harche]

/reopen

[k8s-ci-robot]

@harche: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tallclair]

/remove-lifecycle rotten
/triage accepted

[kannon92]

What is our plans for this KEP in 1.32?

[pacoxu]

What is our plans for this KEP in 1.32?

@kannon92 The blocking issue can be fixed by kubernetes/kubernetes#122778 which is still in review.

Ping @smarterclayton @derekwaynecarr @dchen1107 @SergeyKanzhelev

• If we want to make some progresses on this KEP, we may have to find some resources on the review of Fix evented pleg mirror pod & use IsEventedPLEGInUse instead of FG status check kubernetes#122778 or make another fix for the known issue.

[kannon92]

I'll bring this up in KEP planning.

If you have difficulties getting reviews, I would suggest adding this PR to the sig-node agenda and/or posting on the sig-node slack.

[pacoxu]

We have two blockers here if I understand correctly.

• blocker bugfix [Failing Test] ci-kubernetes-e2e-kind-beta-features/pull-kubernetes-e2e-kind-beta-features failed for enabling some beta features(by default disabled) kubernetes#121349 WIP in : Fix evented pleg mirror pod & use IsEventedPLEGInUse instead of FG status check kubernetes#122778
• performance blocker issue fix EventedPLEG: Pass event created timestamp correctly to cache kubernetes#124297 @christianang

[aojea]

This feature is supposed to replace existing PLEG logic, but it has a track of instability and was also downgraded ... are we going to commit resources to replace existing PLEG logic with this feature?
if not, should we not consider to remove it and once we are equipped start over?