add support for specifying 'compression-level' for NAR's

[dtzWill]

As requested on #2255.

---

Lightly tested, but could probably use a bit more vetting :).

---

May be useful to also add:

• Documentation
• New tests

I'll probably put together at least some test for this,
but help with documentation (what/where/how?) would be appreciated.

---

As an aside, as I was writing the "description" for Setting's that
are part of BinaryCacheStore I wondered:

Are these flags visible somewhere to the user? I couldn't find any way
to get the store-oriented flags to be listed (with or without descriptions/values),
is this currently possible?

[dtzWill]

Rebased to resolve conflicts and work with compressionsink introduction.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-slowdown-handling-large-files-derivations/5310/3

[tomberek]

[tom@tom:~/nix]$ dd if=/dev/urandom of=400MB.rand bs=4096 count=$( expr 100 \* 1024 )
102400+0 records in
102400+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 2.12592 s, 197 MB/s

[tom@tom:~/nix]$ time nix copy --to "file://store?parallel-compression=true" /nix/store/7sy62s
nicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    0m19.810s
user    3m29.199s
sys     0m2.083s

[tom@tom:~/nix]$ time ./result-nix/bin/nix copy --to "file://store?compression-level=0&paralle
l-compression=true" /nix/store/7sy62snicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    0m10.548s
user    2m12.592s
sys     0m0.835s

[tom@tom:~/nix]$ time ./result-nix/bin/nix copy --to "file://store?compression-level=9&parallel-compression=true" /nix/store/7sy62snicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    1m25.031s
user    2m48.845s
sys     0m1.413s

[tom@tom:~/nix]$ time ./result-nix/bin/nix copy --to "file://store?compression-level=6&paralle
l-compression=true" /nix/store/7sy62snicdvfb7g0q5spi1y7gpkay6vz-400MB.rand
warning: dumping very large path (> 256 MiB); this may run out of memory
[1 copied (400.0 MiB)]

real    0m19.636s
user    3m30.792s
sys     0m2.007s

Light testing seems to function as expected.

• compression-level=0&parallel-compression=true
• compression-level=6&parallel-compression=true
• compression-level=1&parallel-compression=false
• compression-level=1&parallel-compression=false&compression=bzip2

Seems to work well. @dtzWill Can this be revived?

Documentation of the various store URI options is still an open issue: #1556

[stale]

I marked this as stale due to inactivity. → More info

[yorickvP]

Still important to me

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/speed-up-nix-copy/15884/3

[tomberek]

Fixed in #5372