Automatic UID allocation #3600

edolstra · 2020-05-20T07:45:59Z

This PR does the following:

It adds an experimental feature and option auto-uid-allocation which provides an alternative to having a nixbld group of pre-created build users. When enabled, Nix allocates UIDs/GIDs in the range 872415232+ on Linux and 56930 on macOS.
It adds an experimental feature cgroups that causes builds to be executed in a cgroup. This allows getting some statistics from a build (such as CPU time) and in the future may allow setting resource limits. But it mainly exists because the uid-range feature requires it.
It adds a system feature uid-range that causes a build to be executed as root in a UID namespace with 65,536 UIDs available. This allows things like systemd-nspawn and NixOS containers to run inside a Nix build.

domenkozar · 2020-05-20T08:00:28Z

A few questions/comments:

Does it work on macOS?
Why does it need to be experimental feature?
Missing documentation for the nix.conf option

7c6f434c · 2020-05-20T09:45:25Z

I think I have heard of people trying to run systemd on NixOS using only cgroup2; and also, of course, I wonder what will be eventually needed to run lighter-weight NixOS tests on non-NixOS Linux. I guess creating a none,name=systemd cgroup hierarchy could be just mentioned in the error message. Is there a simple test I should run on my non-systemd Nixpkgs-kernel system to see if systemd-nspawn indeed works in such setups?

7c6f434c · 2020-05-20T12:27:12Z

auto-allocate-uids needs both to be enabled and added to experimental features; there is a message saying auto-allocate-uids = true is needed, but a more generic message is actually printed.

Once things are configured: I do get «Container nixos exited successfully.», and the target path is built with reasonable output (and there is a running systemd inside the build but no apparent attempts at escaping happen).

Logs: firewall failing seems expected, system-getty.slice failing due to it is slightly surprising, the following might mean I am doing something wrong on the host system or maybe you also get these:

Failed to create symlink /sys/fs/cgroup/cpuacct: Read-only file system
Failed to create symlink /sys/fs/cgroup/cpu: Read-only file system
Failed to create symlink /sys/fs/cgroup/net_prio: Read-only file system
Failed to create symlink /sys/fs/cgroup/net_cls: Read-only file system

Overall nix-store -r time is <3s wall-clock time, which is just great, thanks for that feature.

7c6f434c · 2020-05-20T13:11:38Z

Looked at the code and got an idea to test… auto-allocate-uids requires systemd cgroup hierarchy to exist even if systemd-cgroup feature is not enabled.

edolstra · 2020-05-20T13:14:51Z

auto-allocate-uids requires systemd cgroup hierarchy to exist even if systemd-cgroup feature is not enabled.

Yeah that's currently true, didn't think about that. In principle however it could use any existing hierarchy since it's only used for tracking processes.

7c6f434c · 2020-05-20T13:28:07Z

… or even mount «nix» hierarchy if systemd one is not mounted, I guess…

I am OK with mounting systemd hierarchy on boot, I guess a slightly more detailed message is enough. I just have no cgroup hierarchies mounted by default so it was very cheap for me to check this combination of options.

edolstra · 2020-05-20T13:36:53Z

firewall failing seems expected

Actually that succeeds for me. Which makes me realize a big problem with this approach: whether certain networking features (firewall, NAT, ...) work depend on what kernel modules are loaded in the host system. I don't think there is any way to restrict access...

7c6f434c · 2020-05-20T13:42:49Z

Erm. Does that require structured required features (and, in this instance, assertions about host kernel modules) for a proper-ish solution? I guess whoever cares could write a very minimal sinit VM to demonstrate whether some dependency is not listed…

edolstra · 2020-05-20T13:55:55Z

Or maybe a seccomp filter could be used to restrict access to undeclared features.

7c6f434c · 2020-05-20T14:54:20Z

Seccomp filter sounds like something that requires ahead-of-time enumeration of all possible things that could go wrong

edolstra · 2020-05-20T15:20:34Z

I just realized that the situation isn't that bad. Or rather, it was already bad and this doesn't make it worse. It was already possible to create network devices, firewall tables etc. depending on the host kernel configuration. For example:

with import <nixpkgs> {};

runCommand "foo"
  {
    buildInputs = [ pkgs.utillinux pkgs.iproute pkgs.iptables ];
  }
  ''
    unshare -m -n -U -r -- bash -c "
      set -e
      mkdir -p foo/run foo/nix/store
      mount --rbind /nix/store foo/nix/store
      ip link add foo-h type veth peer name foo-c
      chroot foo iptables -t nat -A POSTROUTING -p tcp -s 192.168.1.2
      chroot foo iptables -t nat -L
    "
    mkdir $out
  ''

This will even cause kernel modules like veth and iptable_nat to be loaded if they're not already.

7c6f434c · 2020-05-20T15:41:41Z

Ah, it might be that the host dependency got exposed by the fact that I did not enable module autoloading. I guess this impurity was «mitigated» by there being little incentive to do such things in public expressions.

Rather than rely on a nixbld group, we now allocate UIDs/GIDs dynamically starting at a configurable ID (872415232 by default). Also, we allocate 2^18 UIDs and GIDs per build, and run the build as root in its UID namespace. (This should not be the default since it breaks some builds. We probably should enable this conditional on a requiredSystemFeature.) The goal is to be able to run (NixOS) containers in a build. However, this will also require some cgroup initialisation. The 2^18 UIDs/GIDs is intended to provide enough ID space to run multiple containers per build, e.g. for distributed NixOS tests.

Also, run builds in a cgroup namespace (ensuring /proc/self/cgroup doesn't leak information about the outside world) and mount /sys. This enables running systemd-nspawn and thus NixOS containers in a Nix build.

2^18 was overkill. The idea was to enable multiple containers to run inside a build. However, those containers can use the same UID range - we don't really care about perfect isolation between containers inside a build.

"uid-range" provides 65536 UIDs to a build and runs the build as root in its user namespace. "systemd-cgroup" allows the build to mount the systemd cgroup controller (needed for running systemd-nspawn and NixOS containers). Also, add a configuration option "auto-allocate-uids" which is needed to enable these features, and some experimental feature gates. So to enable support for containers you need the following in nix.conf: experimental-features = auto-allocate-uids systemd-cgroup auto-allocate-uids = true system-features = uid-range systemd-cgroup

Maybe this should be a separate system feature... /sys exposes a lot of impure info about the host system.

…-allocation

doc/manual/src/release-notes/rl-next.md

src/libstore/globals.hh

nixos-discourse · 2022-11-24T10:48:36Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-11-14-nix-team-meeting-minutes-8/23452/1

Ericson2314 · 2022-11-29T15:05:13Z

src/libstore/build/local-derivation-goal.cc

+       sandbox uids. This must be done before any chownToBuilder()
+       calls. */
+    killSandbox(false);
+
    /* Right platform? */


Any reason this code is no longer at the top? It makes sense to me to check the drv itself before doing IO like checking the filesystem --- cheap checks first.

Ericson2314 · 2022-11-29T15:06:18Z

src/libstore/build/local-derivation-goal.cc

+
+                /* FIXME: set proper permissions in restorePath() so
+                   we don't have to do another traversal. */
+                canonicalisePathMetaData(actualPath, {}, inodesSeen);


This just become needed?

Ericson2314 · 2022-11-29T15:08:02Z

src/libstore/build/local-derivation-goal.cc

@@ -580,10 +642,11 @@ void LocalDerivationGoal::startBuilder()

        printMsg(lvlChatty, format("setting up chroot environment in '%1%'") % chrootRootDir);

-        if (mkdir(chrootRootDir.c_str(), 0750) == -1)
+        // FIXME: make this 0700


This FIXME is fairly short so I doubt I would know how to interpret it a long time from know. Should we file an issue and link it here?

nixos-discourse · 2022-12-06T20:36:27Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-2-12-0-released/23780/4

ncfavier · 2022-12-13T15:51:47Z

This is nice, but it breaks nix-top which gets build users from the nixbld group. Could we have a Nix command that lists the UIDs for the currently active builds? Or does this PR make nix-top obsolete?

edolstra · 2022-12-13T16:02:44Z

@ncfavier Maybe you can create an issue for querying active builds? But maybe it's less important since in principle builds will show up in systemd-cgtop.

nixos-discourse · 2023-04-21T11:57:49Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-team-report-2022-10-2023-03/27486/1

nixos-discourse · 2023-11-20T18:03:04Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-build-ate-my-ram/35752/2

nixos-discourse · 2024-02-23T19:03:38Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-on-macos-now-fails-because-i-set-auto-allocate-uids-true-like-an-idiot/40210/1

This comment was marked as outdated.

Sign in to view

edolstra force-pushed the auto-uid-allocation branch from 90b4689 to e263fd4 Compare June 18, 2020 16:35

edolstra added 9 commits July 6, 2020 13:50

canonicalisePathMetaData(): Support a UID range

c3e0a68

Run builds in their own cgroup

f5fa3de

Also, run builds in a cgroup namespace (ensuring /proc/self/cgroup doesn't leak information about the outside world) and mount /sys. This enables running systemd-nspawn and thus NixOS containers in a Nix build.

Reduce # of UIDs per build to 65536

ca2f64b

2^18 was overkill. The idea was to enable multiple containers to run inside a build. However, those containers can use the same UID range - we don't really care about perfect isolation between containers inside a build.

Destroy the cgroup prior to building

7bdcf43

Simplify cgroup creation

570c443

Fix macOS build

8c4cce5

Only mount /sys in uid-range builds

7349f25

Maybe this should be a separate system feature... /sys exposes a lot of impure info about the host system.

edolstra force-pushed the auto-uid-allocation branch from e263fd4 to 7349f25 Compare July 6, 2020 12:36

edolstra mentioned this pull request Jul 14, 2020

Add nix processes command #3800

Draft

edolstra mentioned this pull request Oct 7, 2020

Split build.cc -- new version of #3098 #4114

Merged

Merge commit 'f66bbd8c7bb1472facf8917e58e3cd4f6ddfa1b5' into auto-uid…

2546c63

…-allocation

github-actions bot requested a review from fricklerhandwerk November 23, 2022 14:25

cole-h reviewed Nov 23, 2022

View reviewed changes

doc/manual/src/release-notes/rl-next.md Outdated Show resolved Hide resolved

src/libstore/globals.hh Show resolved Hide resolved

Include UID in hex

2aa3f2e

edolstra added 6 commits November 27, 2022 16:38

Add tests for auto-uid-allocation, uid-range and cgroups

f1b5c68

Fix evaluation

fc14585

Check that auto-allocated UIDs don't clash with existing accounts

ff12d1c

Add a setting for enabling cgroups

67bcb99

Add example

7dd3e1f

Restore ownership of / for non-uid-range builds

4f762e2

edolstra enabled auto-merge November 29, 2022 12:54

edolstra merged commit fbc53e9 into master Nov 29, 2022

edolstra deleted the auto-uid-allocation branch November 29, 2022 13:01

fricklerhandwerk mentioned this pull request Nov 29, 2022

add CODEOWNERS #7277

Merged

Ericson2314 reviewed Nov 29, 2022

View reviewed changes

ncfavier mentioned this pull request Dec 13, 2022

Allow disabling build users by unsetting build-users-group #7458

Merged

Hoverbear mentioned this pull request Jan 9, 2023

Deleting Users on Mac is not working DeterminateSystems/nix-installer#33

Closed

roberth mentioned this pull request Feb 9, 2023

error: mounting /dev/pts: Operation not permitted #7791

Closed

alyssais mentioned this pull request Feb 12, 2023

Nix 2.12+ allows writing to the sandbox's /etc #7813

Closed

infinisil mentioned this pull request Feb 12, 2024

Utilising the Nix 2.12 features regarding builds tweag/nix-hour#9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatic UID allocation #3600

Automatic UID allocation #3600

edolstra commented May 20, 2020 •

edited

Loading

domenkozar commented May 20, 2020

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

7c6f434c commented May 20, 2020

This comment was marked as outdated.

7c6f434c commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

nixos-discourse commented Nov 24, 2022

Ericson2314 Nov 29, 2022

Ericson2314 Nov 29, 2022

Ericson2314 Nov 29, 2022

nixos-discourse commented Dec 6, 2022

ncfavier commented Dec 13, 2022

edolstra commented Dec 13, 2022

nixos-discourse commented Apr 21, 2023

nixos-discourse commented Nov 20, 2023

nixos-discourse commented Feb 23, 2024

Automatic UID allocation #3600

Automatic UID allocation #3600

Conversation

edolstra commented May 20, 2020 • edited Loading

domenkozar commented May 20, 2020

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

7c6f434c commented May 20, 2020

This comment was marked as outdated.

7c6f434c commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

edolstra commented May 20, 2020

7c6f434c commented May 20, 2020

nixos-discourse commented Nov 24, 2022

Ericson2314 Nov 29, 2022

Choose a reason for hiding this comment

Ericson2314 Nov 29, 2022

Choose a reason for hiding this comment

Ericson2314 Nov 29, 2022

Choose a reason for hiding this comment

nixos-discourse commented Dec 6, 2022

ncfavier commented Dec 13, 2022

edolstra commented Dec 13, 2022

nixos-discourse commented Apr 21, 2023

nixos-discourse commented Nov 20, 2023

nixos-discourse commented Feb 23, 2024

edolstra commented May 20, 2020 •

edited

Loading