stdenv/setup.sh: fix consumeEntire in bash5

[matthewbauer]

Unfortunately, bash5 has changed how read -N 0 works, meaning that
consumeEntire no longer works as expected. Compare:

$ nix shell github:nixos/nixpkgs\?ref=nixos-20.03#bash -c bash -c 'read -N 0 contents < /bin/sh; echo $?'
1
$ nix shell github:nixos/nixpkgs\?ref=nixos-20.03#bash_5 -c bash -c 'read -N 0 contents < /bin/sh; echo $?'
0

Bash 5 no longer reaches EOF when nchars=0. I could not find this
documented anywhere.

The best substitute I can find is to provide the max read nchars
value (2147483647). This appears to have the same effect as nchars=0
in practice.

[abathur]

Not that it really adds much here, but I looked this up in the bash source:

• code itself https://git.savannah.gnu.org/cgit/bash.git/tree/builtins/read.def?h=bash-5.1#n374
• introduced at https://git.savannah.gnu.org/cgit/bash.git/commit/builtins/read.def?h=bash-5.1&id=d233b485e83c3a784b803fb894280773f16f2deb
• changelog entry https://git.savannah.gnu.org/cgit/bash.git/tree/CWRU/changelog#n2306

[worldofpeace]

comment on this magic number perhaps?

[abathur]

I wonder (but have not extensively tested) if you can do something like read -d $'\x04' (source)? Not that you wouldn't still need to comment on a magical value... :)

[andersk]

This doesn’t actually reject null bytes as claimed, nor does it preserve them, since Bash is not capable of storing a null byte in a variable. How about -d ''?

[happysalada]

Please forgive me for coming a little late to the party.
I just gave a go at upgrading to bash5 and ran into this particular issue.
I was trying to understand the reason of existence of consumeEntire wouldn't something like $(<$input) work as well ?
(my understanding is that it just wants to read the content of the file. I don't understand what the problem with null bytes could come from)
I'm going to test running it tonight to see if it changes much.

The other solution that was mentioned in this thread is using the -d ''. I'm not sure how it affects the null byte problem since I don't understand it in the first place.

I understand if you are busy. No need to feel any pressure to reply to this.

Thanks for documenting this in the first place! (I was going insane through the changelogs trying to find why the behavior changed).

[andersk]

@happysalada "$(<"$input")" would strip trailing whitespace (we want to preserve it), and would drop everything after a null byte (we want to at least detect this).

[happysalada]

Thank you for the explanation!
I'm trying out the solution you suggested with the -d ''

The idea behind the nullbyte thing is that perhaps there are hooks provided by users which might contain a null byte ?
Is that correct?
On a first look it seemed to me that we are only getting the input from files that we know about, I couldn't figure out in which case there could be a null byte.

[andersk]

@happysalada This is used in the implementation of substitute, substituteInPlace, substituteAll, and substituteAllInPlace, which in turn are used in the implementation of thousands of packages in nixpkgs and many packages outside nixpkgs. Although it is a mistake to use these on binary files, we want to detect this mistake, not just allow it to silently corrupt your files.

[happysalada]

@andersk Thank you for the clarification!
So the intent is to fail on a binary, it wasn't clear to me just from the function name.
Would something like isutf8 from moreutils be more appropriate ?
I'm not sure it's a great idea to add another dependency, but if it make the intent clearer it might be a good idea.

also if anybody is interested, I'm trying the bash upgrade in this PR #131681
(I'm not saying it will succeed :-) but I want to try)

[andersk]

Would something like isutf8 from moreutils be more appropriate ?

Pure Bash is better, both for performance, and for minimizing unnecessary rebuilds—we wouldn’t want every moreutils upgrade to trigger a full rebuild of every package. And we only care about null bytes, which Bash is unable to handle correctly; we don’t care about invalid UTF-8, which it doesn’t have any particular issues with.

[happysalada]

@andersk Thank you for your answer. It makes sense

Just a small update on this. The bash_5 update is stuck with an error that doesn't make sense to me (one of the nix tests failing). I'm going to try to get some more information on it.
If I fail to do that in a timely manner, I will batch this fix with other proposed shell improvements and submit it to the hydra team for testing, in the hope of getting this merged before the bash_5 upgrade.

[happysalada]

This was just merged to staging in #133495
Thank you again for making this PR and those who reviewed it!