Build failure: nix_2_3 fails in test tests/add.sh sometimes

[ghost]

Steps To Reproduce

when running ofborg eval tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix nix_2_3 fails sometimes. i saw this once in the last staging-next round #292500 (did not keep log) and recently in this PR #292043 (staging commit a8149b5) log.

Build log

https://gist.githubusercontent.com/GrahamcOfBorg/4e0307cf9018f432a637c34bb050c525/raw/bf6ca93a696d44c9a586cc70f499cdb8cdf45036/ofborg-eval-lib-tests

ran test tests/add.sh... [FAIL]
    ++ nix-store --add ./dummy
    + path1=/build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    + echo /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    ++ nix-store --add-fixed sha256 --recursive ./dummy
    + path2=/build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    + echo /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    + test /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy '!=' /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    + echo 'nix-store --add and --add-fixed mismatch'
    nix-store --add and --add-fixed mismatch
    + exit 1
error: builder for '/nix/store/qx8lnz589pjm8wgfks7qnrqs21nqis10-nix-2.3.17.drv' failed with exit code 2;

Notify maintainers

@flokli @RaitoBezarius

Metadata

ofborg eval machine.

Add a 👍 reaction to issues you find important.

[ghost]

seen in #294068 (comment) as well

[ghost]

failed again. https://gist.github.com/GrahamcOfBorg/e78ff150d9ee69e938f31646bba6893d
staging change e5f37c0 my change 49bccc2

ran test tests/add.sh... [FAIL]
    ++ nix-store --add ./dummy
    + path1=/build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    + echo /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    ++ nix-store --add-fixed sha256 --recursive ./dummy
    + path2=/build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    + echo /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    + test /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy '!=' /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    ++ nix-store --add-fixed sha256 ./dummy
    + path3=/build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    + echo /build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    /build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    + test /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy '!=' /build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    ++ nix-store --add-fixed sha1 --recursive ./dummy
    + path4=/build/nix-test/add/store/dcjypnx18bpzy81n499bfgq7fl548swl-dummy
    + echo /build/nix-test/add/store/dcjypnx18bpzy81n499bfgq7fl548swl-dummy
    /build/nix-test/add/store/dcjypnx18bpzy81n499bfgq7fl548swl-dummy
    + test /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy '!=' /build/nix-test/add/store/dcjypnx18bpzy81n499bfgq7fl548swl-dummy
    ++ nix-store -q --hash /build/nix-test/add/store/g1qxg63zbanhv79ibby90311521d4237-dummy
    + hash1=sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p
    + echo sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p
    sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p
    ++ nix-hash --type sha256 --base32 ./dummy
    + hash2=0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw
    + echo 0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw
    0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw
    + test sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p = sha256:0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw
make: *** [mk/lib.mk:128: tests/add.sh.test] Error 1
make: *** Waiting for unfinished jobs....

[flokli]

This is flakiness in the upstream Nix test suite.

I heard it got better, as in, there's been some flakiness fixes, but those were not backported to 2.3 yet. Someone needs to dig into this to get it less flaky.

[ghost]

my suspicion is that the tests are run in parallel and there is some test that writes to the file that add.sh is using.

$   grep dummy **/*(.)
add.sh:path1=$(nix-store --add ./dummy)
add.sh:path2=$(nix-store --add-fixed sha256 --recursive ./dummy)
add.sh:path3=$(nix-store --add-fixed sha256 ./dummy)
add.sh:path4=$(nix-store --add-fixed sha1 --recursive ./dummy)
add.sh:hash2=$(nix-hash --type sha256 --base32 ./dummy)
check.nix:    url = "file://" + toString ./dummy;
check.sh:echo -n > ./dummy
check.sh:echo 'Hello World' > ./dummy
fixed.builder2.sh:echo dummy: $dummy
fixed.builder2.sh:if test -n "$dummy"; then sleep 2; fi
fixed.nix:  f2 = dummy: builder: mode: algo: hash: mkDerivation {
fixed.nix:    inherit dummy;
hash-check.nix:    args = ["-e" "-x" ./dummy];
hash-check.nix:    args = ["-e" "-x" ./dummy];
hash-check.nix:    args = ["-e" "-x" (./dummy  + "/FOOBAR/../.")];
init.sh:echo 'Hello World' > ./dummy
lang/eval-okay-eq-derivations.nix:  drvA3 = derivation { name = "a"; builder = "/foo"; system = "i686-linux"; } // { dummy = 1; };
user-envs.nix:# Some dummy arguments...

so perhaps check.sh is writing to dummy while add.sh is running.

i tried to enter the builder using nix-shell ../../src/nix.4 -A nix_2_3 --pure then running genericBuild but it fails to build.

[ghost]

until the problem is fixed perhaps disabling the test can be disabled? i seem to hit it frequently when doing evals on staging.

created PR #294932 to disable parallel checking.

[flokli]

Worked around by #294932, upstream issue NixOS/nix#10222