Support User Namespaces in pods

[derekwaynecarr]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add support for user namespaces in pods
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/README.md
• Discussion Link: This was discussed in several sig-node meetings, most notable Nov 2 2021 and in this PR, that feedback was incorporated already by this KEP is taking: keps/127: Support User Namespaces #2101.
• Primary contact (assignee): @rata @giuseppe
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.25
  • Beta release target (x.y): 1.30
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • v1.24 127: Add KEP for user namespaces support #3065
    • v1.25 KEP-127: Mark as implementable (target phase I for 1.25) #3275
    • v1.27 KEP-127: Support userns in stateless pods with idmap mounts #3811
    • v1.28 [KEP-127] Add Pod Security Standards to User Namespaces KEP #4044
    • v1.28 KEP-127: add support for stateful pods #4084
    • v1.28 KEP-127: Fix user namespaces feature gate name #4107
    • v1.28 KEP-127: Update feature gate name and update last-milestone for 1.29 #4147
  • Code (k/k) update PR(s):
    • v1.27 KEP-127: user namespace support for stateless pods kubernetes#116377
    • v1.28 apis: drop check for volumes with user namespaces kubernetes#118691
    • v1.29 KEP-127: Update PSS based on feature gate kubernetes#118760
  • Docs (k/website) update PR(s):
    • v1.27 Document user namespace changes in v1.27 (KEP-127) website#39860
    • v1.28 Doc update for userns in 1.28 website#41908 (v1.28)
    • v1.29 [KEP-127] Add documentation about user namespaces and PSS website#43803 (v1.29)
• Beta
  • KEP (k/enhancements) update PR(s): KEP-127: graduate to Beta for 1.30 #4439
  • Code (k/k) update PR(s):
    • KEP-127: require userns support from the CRI runtime before using it kubernetes#123216
    • KEP-127: kubelet: honor kubelet user mappings kubernetes#123593
    • KEP-127: Add UserNamespacesPodSecurityStandards e2e test kubernetes#121606
  • Docs (k/website) update(s):
    • Feature blog 1.30: user namespaces website#45354
    • User namespaces doc changes for 1.30 website#45178

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[derekwaynecarr]

This work is being done by @pweil- and is reviewed by @derekwaynecarr, it is sponsored by @kubernetes/sig-node

[mdshuai]

@derekwaynecarr Could you help create a user story card for this feature?

[idvoretskyi]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

[pweil-]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

Yes, this feature is experimental only so it would be considered alpha.

[idvoretskyi]

@derekwaynecarr @pweil- can you confirm that this item targets beta in 1.6?

[adelton]

@derekwaynecarr, the proposal kubernetes/kubernetes#34569 was closed by bot due to inactivity.

@pweil-, in kubernetes/kubernetes#34569 (comment) you've proposed the approach pweil-/kubernetes@16f29eb which changes the group of /var/lib/kubelet/pods to the remapped root group. Do I understand it correctly that this is currently not tracked in any pull request?

[adelton]

@pweil-, I also wonder if similar to docker's /var/lib/docker/<uid>.<gid> approach when --userns-remap is used, it might make sense to use /var/lib/kubelet/pods-<uid>.<gid> and just chown/chgroup everything in those subdirectories to the remapped <uid>.<gid>. Why did you opt for just the chgrp and not the full chown?

[pweil-]

@adelton in the end, I think having this be transparent to Kubernetes is the right approach. Whether that be something like shiftfs or implementation in the CRI (moby/moby#28593). You are correct that my existing proposal is not currently tracked in an open PR anymore.

The reasoning behind using the chgrp was to follow our fsgroup strategy where we just ensure group access instead of uid access.

[adelton]

Thanks @pweil-.

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

I have now filed kubernetes/kubernetes#55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

[pweil-]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question 😄

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

Yes

I have now filed kubernetes/kubernetes#55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

👍 subscribed

[adelton]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question

Ideally, the pod would specify how many distinct uids/gids it would require / list of uids it wants to see inside of the containers, and docker or different container runtime would setup the user namespace accordingly. But unless docker also changes ownership of the volumes mounted to the containers, Kubernetes will have to do that as part of the setup.

[adelton]

@pwel-, what is the best way to get some review and comments on kubernetes/kubernetes#55707, to get it closer to mergeable state?

[0xmichalis]

@pweil- ^

[pweil-]

@adelton I would try to engage the sig-node folks either at their Tuesday meeting or on slack: https://github.com/kubernetes/community/tree/master/sig-node

[adelton]

@derekwaynecarr, could you please bring kubernetes/kubernetes#55707 to sig-node's radar?

[idvoretskyi]

@pweil- @derekwaynecarr any progress on this feature is expected?

[drewhagen]

Hello @derekwaynecarr 👋, 1.30 Docs Lead here.

Does this enhancement work planned for 1.30 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.30 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday February 22nd 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[rata]

@Checksumz thanks! I'm interested in writing a blog post for the 1.30 release. Let me confirm during next week if I'll do it. I want to check some stuff with other project releases.

@drewhagen yes, there are doc changes on this release. I've just opened a placeholder PR: kubernetes/website#45178. Thanks for the remainder!

[Checksumz]

@Checksumz thanks! I'm interested in writing a blog post for the 1.30 release. Let me confirm during next week if I'll do it. I want to check some stuff with other project releases.

@drewhagen yes, there are doc changes on this release. I've just opened a placeholder PR: kubernetes/website#45178. Thanks for the remainder!

thanks for your response @rata

[tjons]

Hey again @derekwaynecarr 👋 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 6th March 2024 .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PR is open and needs to be merged before code freeze:

• KEP-127: require userns support from the CRI runtime before using it kubernetes#123216

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

[Checksumz]

@Checksumz thanks! I'm interested in writing a blog post for the 1.30 release. Let me confirm during next week if I'll do it. I want to check some stuff with other project releases.

@drewhagen yes, there are doc changes on this release. I've just opened a placeholder PR: kubernetes/website#45178. Thanks for the remainder!

@rata Thank you for your interest in the 1.30 Feature Blog please can you open a placeholder PR by February 26th ?

[rata]

@Checksumz Here it is (against dev-1.30): kubernetes/website#45354

Thanks!

[tjons]

Hello @derekwaynecarr 👋, Enhancements team here.

With all the implementation(code related) PRs merged as per the issue description:

• KEP-127: require userns support from the CRI runtime before using it kubernetes#123216

This enhancement is now marked as tracked for code freeze for the 1.30 Code Freeze!

[sreeram-venkitesh]

Hello 👋, 1.31 Enhancements Lead here.

If you wish to progress this enhancement in v1.31, please have the SIG lead opt-in your enhancement by adding the lead-opted-in label and set the milestone to v1.31 before the Production Readiness Review Freeze.

/remove-label lead-opted-in

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[remram44]

/remove-lifecycle stale

[kannon92]

What are the plans for 1.32?

[rata]

No plans for 1.32 so far, just trying to get final releases of runc and so to use this. We will be back afterwards :)

[utam0k]

No plans for 1.32 so far, just trying to get final releases of runc and so to use this. We will be back afterwards :)

@rata runc and containerd released the version having support for UN and id mapped mount. And, I realized that NFS haven't support NFS id-mapped mount yet. Should we take care of it in the documentation?

[rata]

@utam0k We should link to the mount_setattr manpage, that list the supported fs and the kernel that added support for it. I send a patch to update it a few months ago, I think it is still up to date. The manpages was not updating the website often, but now it is. Wanna do a doc patch to link to this page https://man7.org/linux/man-pages/man2/mount_setattr.2.html ?

[utam0k]

@rata I've updated my commend because I made a mistake.
#127 (comment)

Wanna do a doc patch to link to this page man7.org/linux/man-pages/man2/mount_setattr.2.html ?

hmm... It's one of the good ideas, but out-of-tree fs, e.g., zfs doesn't list on the man page.
How about adding a note about this concern and encouraging users to refer to the man page?

[utam0k]

No plans for 1.32 so far, just trying to get final releases of runc and so to use this. We will be back afterwards :)

@rata Are you aiming for stable at 1.33? Please let me know if there is anything I can do to help.

[utam0k]

I don't have a good solution, but I'll write down one concern. I think there will be a security problem if a filesystem that doesn't support id-mapped mounts ignores the option and mounts it. NFS doesn't support id-mapped mounts, but it returns an error properly, so the Pod will fail to start. It is not known whether it is worth warning about in the user documentation.
https://elixir.bootlin.com/linux/v6.12/source/fs/nfsd/export.c#L398-L450

However, I am a beginner in the filesystem, so please let me know if there are any mistakes.