Kubelet OpenTelemetry Tracing

[sallyom]

Enhancement Description

• One-line enhancement description (can be used as a release note): Instrument kubelet to generate and export OpenTelemetry trace data.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/2831-kubelet-tracing/README.md
• Discussion Link:
• Primary contacts (assignee): @somalley, @dashpole
• Responsible SIGs: SIG-Instrumentation
• Enhancement target (which target equals to which milestone):
  • Alpha release target (1.25)
  • Beta release target (1.27)
  • Stable release target (?)
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-2831: Instrumenting Kubelet for OpenTelemetry Tracing #2832
  • Code (k/k) update PR(s):
    • kubelet tracing instrumentation kubernetes#105126
  • Docs (k/website) update PR(s):
    • Add docs for KubeletTracing feature gate 1.25 website#35137
• Beta
  • KEP (k/enhancements) update PR(s):
    • KEP-2831: adding beta graduation criteria #3714
  • Code (k/k) update PR(s):
    • kubelet: create top-level traces for pod sync and GC kubernetes#114504
    • Graduate KubeletTracing feature to beta kubernetes#115750
  • Docs (k/website) update(s):
    • [KEP-2731] Add docs for Kubelet OpenTelemetry tracing graduation website#39904
• Stable
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[kikisdeliveryservice]

@sallyom can you please add the mandatory discussion link to this issue?

[ehashman]

/sig node instrumentation

[logicalhan]

/assign

[shivanshuraj1333]

/cc

[yangjunmyfm192085]

/cc

[249043822]

/cc

[hosseinsalahi]

Hello @sallyom

v1.24 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd, 2022. This enhancement is targeting alpha for v1.24,

Here’s where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo.
  - KEP file PR is pending.
• KEP status is marked as implementable for this release
• KEP has a test plan section filled out.
• KEP has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date for release v1.24.
Thanks!

[sallyom]

@encodeflush thanks for the check-in - I'm not sure what the production readiness review is edit: found it, will add the questionnaire, I don't see it on the enhancements README can you point me to the steps to get that done? Thanks again. Also, am I meant to check those off as they are done or are they for someone else to decide?

[hosseinsalahi]

@sallyom Thanks for the update. Those criterion should be fulfilled form your side. From enhancements team point of view, we go through them and check if your KEP meets the requirements.

[chrisnegus]

Hi @sallyom 👋 1.24 Docs shadow here.

This enhancement is marked as 'Needs Docs' for the 1.24 release.

Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu March 31, 11:59 PM PDT.

Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thanks!

[valaparthvi]

Hi @sallyom 👋 1.24 Release Comms team here.

We have an opt-in process for the feature blog delivery. If you would like to publish a feature blog for this issue in this cycle, then please opt in on this tracking sheet.

The deadline for submissions and the feature blog freeze is scheduled for 01:00 UTC Wednesday 23rd March 2022 / 18:00 PDT Tuesday 22nd March 2022. Other important dates for delivery and review are listed here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24#timeline.

For reference, here is the blog for 1.23.

Please feel free to reach out any time to me or on the #release-comms channel with questions or comments.

Thanks!

[hosseinsalahi]

Hello @sallyom

I'm just checking in once more as we approach the 1.24 Code Freeze on 18:00 PDT, Tuesday, March 29th 2022

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are merged by the code freeze deadline.
• Have a documentation placeholder PR open by 18:00 PDT, Thursday, March 31, 2022.

For note, the status of this enhancement is currently marked as at risk.

Could you please confirm if following PR is part of the implementation for this enhancement?

• kubelet tracing instrumentation kubernetes#105126
• KEP-3085: Add KEP for Pod network ready condition #3087

Kindly please let me know if I'm missing any related PRs other than the ones I linked above. Thank you so much!

[ddebroy]

#3087 is not directly related to the overall OpenTelemetry enhancements.

[sallyom]

kubernetes/kubernetes#105126 is the initial implementation for this enhancement

[valaparthvi]

Hi @sallyom 👋 1.24 Release Comms team here.

We have an opt-in process for the feature blog delivery. If you would like to publish a feature blog for this issue in this cycle, then please opt in on this tracking sheet.

The deadline for submissions and the feature blog freeze is scheduled for 01:00 UTC Wednesday 23rd March 2022 / 18:00 PDT Tuesday 22nd March 2022. Other important dates for delivery and review are listed here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24#timeline.

For reference, here is the blog for 1.23.

Please feel free to reach out any time to me or on the #release-comms channel with questions or comments.

Thanks!

Hey @sallyom, can you please add a placeholder PR for this feature in https://github.com/kubernetes/website by March 30, that is our extended deadline for feature blog freeze?

Example of a feature blog PR: kubernetes/website#30538
Example of a feature blog: https://github.com/kubernetes/website/blob/main/content/en/blog/_posts/2021-12-08-dual-stack-networking-ga.md

[gracenng]

Hi, 1.24 Enhancements Lead here 👋. With code freeze now in effect, this enhancement has not met the criteria for the freeze and has been removed from the milestone.

As a reminder, the criteria for code freeze is:

All PRs to the kubernetes/kubernetes repo have merged by the code freeze deadline
Feel free to file an exception to add this back to the release. If you plan to do so, please file this as early as possible.

Thanks!
/milestone clear

[kannon92]

1.32 is coming up soon so please let us know if you want to graduate this to stable.

[haircommander]

Having not heard anything in two weeks, I've moved this as not planned for 1.32. Please comment if you disagree

[dashpole]

I might have time to graduate this (need to double-check the criterion), but we can add it to the milestone in the meantime.

[SergeyKanzhelev]

Moved to proposed so we can discuss who can be revioewer/approver from SIG Node side.

[haircommander]

/label lead-opted-in

[SergeyKanzhelev]

I might have time to graduate this (need to double-check the criterion), but we can add it to the milestone in the meantime.

@dashpole please ping me on PRs if you plan to move this one forward

[tjons]

Hello @dashpole 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage stable for v1.32 (correct me, if otherwise).

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday, October 3rd, 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP has a production readiness review that has been completed and merged into k/enhancements.
• Add GA graduation criteria to the KEP
• Update the KEP to use the latest template. There are several required sections missing: Risks and Mitigations, Upgrade/Downgrade Strategy, and Version Skew Strategy. Please ensure that all required sections (not marked Optional in https://github.com/kubernetes/enhancements/blob/master/keps/NNNN-kep-template/README.md`) are present.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[ffromani]

Hello! I'm among the KEP wranglers that sig-node set up to help sig-node KEPs make progress smoothly during the 1.32 timeframe. Hello @dashpole (cc @SergeyKanzhelev ) asking about #2831 (comment) do you think you will have capacity to work on this enhancement in the 1.32 cycle? Thanks!

[ffromani]

Sorry I forgot previously: PRR freeze on 3rd Oct! If you plan to move forward please post an update for this KEP so you opt in for PRR review! If you plan to move forward please post an update so you opt-in in the PRR review! @sallyom @dashpole

[ffromani]

ping @sallyom @dashpole about opting in the PRR review because the deadline is looming

[mattbailey]

Hello, I'd like to shadow PRR review here, thanks!

Edit: doesn't look like Stable PRR has been created, will keep an eye here.

[haircommander]

/remove-milestone v1.32
/remove-label lead-opted-in

Today's enhancement freeze and I don't see a PR to move this to stable. I am removing this from this release, please let me know if this is a mistake

[dipesh-rawat]

Hello @dashpole @haircommander 👋, 1.32 Enhancements team here.

Since this enhancement has been rescheduled for a future release, it's now marked as Deferred for the current v1.32 release cycle. Confirming this based on the comment #2831 (comment). Thanks!

/milestone clear

[dashpole]

I went through the kubelet codebase, and found the following places we could potentially add tracing:

• podresources grpc server
• CSI grpc client
• devicemanager client/server
• DRA client
• Pluginmanager client

I don't think any of those should block GA.

The only remaining step is to graduate the configuration resource (staging/src/k8s.io/apiserver/pkg/apis/apiserver/v1beta1/types.go).

[saschagrunert]

I don't think any of those should block GA.

I agree. Would that also mean that we graduate to GA in the first place and are free to add tracing later on per demand?

[dashpole]

I agree. Would that also mean that we graduate to GA in the first place and are free to add tracing later on per demand?

Yes, I think that is the right approach.