nix-build: requires non-existent output 'out' from input derivation

[Izorkin]

Describe the bug

Packages build fails very often:

checking for true... true
error (ignored): error: cannot unlink '/tmp/nix-build-python3-minimal-3.9.12.drv-1/Python-3.9.12': Directory not empty
error: derivation '/nix/store/6xjmvr03428342gs1432ac0mn8myxnpc-grub-config.xml.drv' requires non-existent output 'out' from input derivation '/nix/store/88vs9d74w69j14s611vb5v7cn0
...
ran test tests/config.sh... [PASS]
installCheckPhase completed in 40 seconds
error: derivation '/nix/store/cqmxz06ggrnhr7wzf6s1kf39hykmsh6g-local-cmds.drv' requires non-existent output 'out' from input derivation '/nix/store/dw0x90b23k67mkav4bf24r63nvn97dir-nix-2.8.1.drv'
...
error: derivation '/nix/store/shnrbk0l8qdsfmvf4x6qp6nlvc0kdfjc-libmodsecurity-3.0.6.drv' requires non-existent output 'out' from input derivation '/nix/store/nrpsq5ld8fjp3adpimdiwkmg7wbhqsq3-lmdb-0.9.29.drv'
...
checking for references to /build/ in /nix/store/90wv2xdvci79wnj8nxlrxy73j1hiksjh-bind-9.18.3-host...
error: derivation '/nix/store/irc43zk5n8w2bp7ig0d5hlfi4ybm3vs9-system-path.drv' requires non-existent output 'host' from input derivation '/nix/store/ll61r00n2f62qkxlll7y0ba45mzby2m4-bind-9.18.3.drv'

I have to restart the build process manually each time.
The problem happens on nix versions 2.8.1 and 2.9.0pre20220512_d354fc3
On version 2.8.0 the problem rarely occurred

[milahu]

I have to restart the build process

same here with
nix (Nix) 2.9.0pre20220505_f4102de and
nix (Nix) 2.9.1

i have to call nix-build . -A qt6.qtwebengine 3 times to fix

error: derivation qtpositioning requires non-existent output 'out' from input derivation qtserialport

this is after nix garbage collect = empty store. no problem with populated store

build log

nix-build.qt6.qtwebengine.nix-bug-6572.log

input: asjckii7hs9mpm3f623vyhb11hk9qiv6-source == vg85l7pk2cphsb7wbvp4j29xsa449ji8-source.drv
build: bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv

relevant logs

$ nix-build . -A qt6.qtwebengine -vvvv

checking substituter 'https://cache.nixos.org' for path '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source'
checking substituter 'https://cache.nixos.org' for path '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source'

substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source': created
substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source': woken up

building of '/nix/store/bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv!out' from .drv file: waitee 'building of '/nix/store/426q49fmidvdqgfjd68mpr1sl1clvwyd-xkeyboard-config-2.33.drv!out' from .drv file' done; 3 left
building of '/nix/store/bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv!out' from .drv file: waitee 'building of '/nix/store/syz2smdcik9b1asd1pbcddbz5pgzqqm9-xrandr-1.5.1.drv!out' from .drv file' done; 2 left
building of '/nix/store/bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv!out' from .drv file: waitee 'building of '/nix/store/l25hhx6c3jdgkh2vi58k3mf2fbvs0377-zlib-1.2.12.drv!dev' from .drv file' done; 1 left

substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source': init
substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source': trying next substituter
substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source': trying next substituter
path '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source' is required, but there is no substituter that can build it
substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source': done

building of '/nix/store/vg85l7pk2cphsb7wbvp4j29xsa449ji8-source.drv!out' from .drv file: waitee 'substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source'' done; 0 left

substitution of '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source': goal destroyed
locking path '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source'
lock acquired on '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source.lock'
removing invalid path '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source'

setting builder env variable 'out'='/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source'
scanning for references for output 'out' in temp location '/nix/store/vg85l7pk2cphsb7wbvp4j29xsa449ji8-source.drv.chroot/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source'

lock released on '/nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source.lock'

building of '/nix/store/bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv!out' from .drv file: waitee 'building of '/nix/store/vg85l7pk2cphsb7wbvp4j29xsa449ji8-source.drv!out' from .drv file' done; 0 left

building of '/nix/store/bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv!out' from .drv file: all inputs realised
building of '/nix/store/bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv!out' from .drv file: trying to build

error: builder for '/nix/store/bj4f7h30ak3k65b3n3zidwyr6gcbsbdr-qtwebengine-6.3.1.drv' failed with exit code 1;
> build input /nix/store/asjckii7hs9mpm3f623vyhb11hk9qiv6-source does not exist

the "removing invalid path" looks suspicious

[NickCao]

My guess would be nix starting the next build too early before the outputs of the previous build is properly registered.

[milahu]

aka build input X does not exist #1744

in something simple as

{ pkgs }:
with pkgs;
let
  x = fetchFromGitHub {
    # ...
  };
in
stdenv.mkDerivation {
  buildInputs = [ x ];
  # ...
}

this time, restarting the build does not help

i have to manually eval the first derivation to fix this

qtwebengine.nix

these 2 derivations will be built:
  /nix/store/g60qiv2b65238a6f4x32abm4n25028a1-source.drv
  /nix/store/dvblz3fi0x3227kj393y0vcmkann8rg6-qtwebengine-6.3.1.drv
building '/nix/store/g60qiv2b65238a6f4x32abm4n25028a1-source.drv'...

trying https://github.com/milahu/gnumake-tokenpool/archive/c85ddf72c69ccea41f840e5fcfa58ee6719aa038.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  103k    0  103k    0     0   140k      0 --:--:-- --:--:-- --:--:--  140k
unpacking source archive /build/c85ddf72c69ccea41f840e5fcfa58ee6719aa038.tar.gz
building '/nix/store/dvblz3fi0x3227kj393y0vcmkann8rg6-qtwebengine-6.3.1.drv'...
build input /nix/store/l2vvz50anjhw7g1l9xkzjcs5k71nqw11-source does not exist
error: builder for '/nix/store/dvblz3fi0x3227kj393y0vcmkann8rg6-qtwebengine-6.3.1.drv' failed with exit code 1;
       last 1 log lines:
       > build input /nix/store/l2vvz50anjhw7g1l9xkzjcs5k71nqw11-source does not exist
       For full logs, run 'nix log /nix/store/dvblz3fi0x3227kj393y0vcmkann8rg6-qtwebengine-6.3.1.drv'.

nix-build -E 'with import <nixpkgs> {}; with pkgs; fetchFromGitHub {
     owner = "milahu"; 
     repo = "gnumake-tokenpool";
     rev = "5dabefe12144bb91b3bf9b8ec67004282e6f0f18";
     sha256 = "6CfKYQgYSzR/8Nule7gWkwn0W9crqlWcgPIr39LUYsk=";
   }'

this works as expected

nix-build -E 'with import <nixpkgs> {}; with pkgs; let x = stdenv.mkDerivation { name = "x"; unpackPhase = "sleep 999"; }; in stdenv.mkDerivation { name = "y"; buildInputs = [ x ]; }'

aka do not know how to unpack source archive

unpacking sources
unpacking source archive /nix/store/2nwk707f9w8ydnqh10hwq54bxdm3cgl4-source
do not know how to unpack source archive /nix/store/2nwk707f9w8ydnqh10hwq54bxdm3cgl4-source

$ ls /nix/store/2nwk707f9w8ydnqh10hwq54bxdm3cgl4-source
ls: cannot access '/nix/store/2nwk707f9w8ydnqh10hwq54bxdm3cgl4-source': No such file or directory

... where source is a fetchFromGitHub

[Kha]

This issue is wreaking havoc on our CI. Fortunately it turns out that it is at least perfectly reproducible there, so I was able to bisect it using the Hydra binary tarballs down to the merge of #6227 /cc @edolstra. Which at least seems plausible as that PR directly changed the conditional leading to the error: https://github.com/NixOS/nix/pull/6227/files#diff-5784d1ad7c22601c75b86872534049bea5173377de5d61b7446452beabc16d3fL491-R534

[Artturin]

most nixpkgs staging PRs have this issue in ofborg-eval-lib-tests :/

[ghost]

I am suffering from this too and downgraded to nix 2.7.

[colemickens]

Hopefully this is somewhat helpful, but I'm experiencing this (1) seemingly more lately, (2) almost every time I have a large cross-compile rebuild. Not sure if it's the cross-compile part, the large build part, or if it's a misnomer. I know that when I cross rebuild my risc-v config, I basically need to instead run until ./build-that.sh; do sleep 1; done to get it to actually keep progressing and eventually finish the build.

edit: my current nixbuild.sh wrapper looks like this: (it retries for this error, "signal 9", etc): https://github.com/colemickens/nixcfg/blob/bb78cb9e42d4a78d963548e52c948aaf781292e5/misc/nixbuild.sh#L48-L74

[kjeremy]

I am suffering from this too and downgraded to nix 2.7.

Same here but now I hit #6816 (which I can patch around) and #6253.

[ghost]

Edit: I guess while true; do nixos-install; done is a solution for now.

[endgame]

Edit: I guess while true; do nixos-install; done is a solution for now.

Not completely. I sometimes like to stress-test my computer and nixpkgs by running nixos-rebuild --no-net, and after looping for a while I was unable to build anything really because any build would fail with "too many open files". So I suspect when this bug is triggered and the build is aborted, Nix is not closing open files properly.

EDIT: Disregard, I can trigger this from a cold boot.

[kjeremy]

We hit this in CI where looping forever is not an option.

…

On Sun, Oct 2, 2022, 10:33 PM endgame ***@***.***> wrote: Edit: I guess while true; do nixos-install; done is a solution for now. Not completely. I sometimes like to stress-test my computer and nixpkgs by running nixos-rebuild --no-net, and after looping for a while I was unable to build anything really because any build would fail with "too many open files". So I suspect when this bug is triggered and the build is aborted, Nix is not closing open files properly. — Reply to this email directly, view it on GitHub <#6572 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBACRF7FXIDP5MLUQOU4ODWBJAX3ANCNFSM5W6EUWFA> . You are receiving this because you commented.Message ID: ***@***.***>

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/content-addressed-nix-call-for-testers/12881/206

[Ericson2314]

My rough guess is that this is one of the many bugs that arose with the implementation of impure derivations #6227

[endgame]

@Profpatsch because people's understanding has been slowly evolving over those six months. I mean, it was only eight days ago that @Kha posted a very good theory about what's really going on.

Also note that older Nix is technically affected, but it's much harder to trigger.

[Profpatsch]

I get the feeling it was not seen as very high priority.

[Profpatsch]

I revived nix 2.6 in nixpkgs NixOS/nixpkgs#200357

[roberth]

It seems that a dependency between the multi-outputs is required: #7283
Outputs that are independent store paths do not cause the error.

[K900]

🎉