Builds using Go from nixpkgs are not bitwise identical to builds using official Go releases

[tie]

Describe the bug

Programs built using Go 1.16.4 from nixpkgs are not bitwise identical to builds using official Go releases.

To Reproduce

Run the following steps on macOS, some Linux distro, Windows, etc. with Go 1.16.4 installed from official releases. E.g. download using

go install golang.org/dl/go1.16.4@v0.0.0-20210506185525-b8dea299038d
go1.16.4 download
export GOROOT=$HOME/sdk/go1.16.4
export PATH=$GOROOT/bin:$PATH

Then repeat the same steps using Go 1.16.4 from nixpkgs.

1. Ensure that Go builds are reproducible

   # Disable external tool invocations (e.g. gcc and ld).
   export CGO_ENABLED=0 GO_EXTLINK_ENABLED=0
   # Trim absolute paths.
   export GOFLAGS=-trimpath
   # Target the same OS and arch.
   export GOOS=linux GOARCH=amd64
   

2. Set up a simple Go module.

   mkdir nixpkgs-reproduce && cd nixpkgs-reproduce
   go mod init example.com/main
   cat >main.go <<EOF
   package main
   
   func main() {
   	println("hello world")
   }
   EOF
   

3. Run go build and verify checksum.

   go build -o main
   shasum main
   

Expected behavior

SHA sum is the same across all machines. Installing Go using Nix should not change that.

873247e1d242ff1e61abdcd7ce40bf91ea379187  main

Actual behavior

When using Go 1.16.4 from nixpkgs the SHA sum is different.

e204421146b8f1c50af743694370625cca773a3e  main

251,254c251,254
< 00000fa0: 5300 0000 0400 0000 476f 0000 5851 7678  S.......Go..XQvx
< 00000fb0: 6e37 2d65 514b 4472 7a5a 3171 6a36 3950  n7-eQKDrzZ1qj69P
< 00000fc0: 2f42 6533 6459 7538 3148 5378 6538 6668  /Be3dYu81HSxe8fh
< 00000fd0: 516e 5356 572f 4335 736b 7347 794e 5533  QnSVW/C5sksGyNU3
---
> 00000fa0: 5300 0000 0400 0000 476f 0000 4c59 5153  S.......Go..LYQS
> 00000fb0: 486a 3754 3250 4533 5573 4848 6f63 5359  Hj7T2PE3UsHHocSY
> 00000fc0: 2f4b 3676 7342 736d 736d 4f72 4631 666e  /K6vsBsmsmOrF1fn
> 00000fd0: 544b 4445 462f 4335 736b 7347 794e 5533  TKDEF/C5sksGyNU3

At least for a simple “hello world” only Go Build ID is different. Edit: looking at the patches, that’s not limited to just build ID since there are some minor changes in standard library.

See also golang/go#34186 and https://github.com/golang/go/blob/3b770f2ccb1fa6fecc22ea822a19447b10b70c5c/src/cmd/go/internal/work/buildid.go#L22-L93

Notify maintainers

Please @ people who are in the meta.maintainers list of the offending package or module.
If in doubt, check git blame for whoever last touched something.

I don’t think I should ping everyone in teams.golang.members.

[sarahhodne]

I just noticed this today as well, trying to get reproducible builds with Go working. I can reproduce with nixpkgs 6933d06, using the go package (go1.16.4).

[tie]

To be fair, this issue doesn’t break build reproducibility per se but ignores that the rest of the world uses official Go releases. So it’s almost impossible to reproduce Nixpkgs-Go-built binaries using standard toolchain and vice versa.

That said, a gentle ping to @zowoq since you are the last committer.

[tie]

I think the best course would be to leave go1.16 in nixpkgs as is and instead start using the official builds for the upcoming go1.17 release.

[endocrimes]

Unfortunately some of the patches that we apply aren't really well documented (https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/go/1.16.nix), specifically things like the SSL Cert patching - I'll go history diving soon if nobody gets to it before me - hopefully we can move closer to upstream source at least.

[wilsonehusin]

I'm doing a quick read through of patches made on non-test files and I think I only see 2 things:

• MIME type

  nixpkgs/pkgs/development/compilers/go/2-dev.nix

  Line 81 in 093aa2f

  sed -i 's,\"/etc/mime.types,"${mailcap}/etc/mime.types\"\,\n\t&,' src/mime/type_unix.go

• net/{lookup,port}

  nixpkgs/pkgs/development/compilers/go/2-dev.nix

  Lines 121 to 122 in 384d9fe

  	sed -i 's,/etc/protocols,${iana-etc}/etc/protocols,' src/net/lookup_unix.go
  	sed -i 's,/etc/services,${iana-etc}/etc/services,' src/net/port_unix.go

• SSL certs (per @endocrimes above)

I went through history and seems like the first ever SSL patching were done in 1.7, which is understandable as upstream hard-coded the paths.

However, starting 1.9, upstream supports overriding the paths through SSL_CERT_{FILE,DIR} and still does on HEAD. At glance, this seems to be sufficient for what we need, though I have not made attempts so feel free to go ahead if anyone has picked up interest in this.

As for MIME Type and net/{lookup,port}, I don't see immediate solution from upstream to remove our path patching, so maybe we can propose to golang upstream for a configurable approach?

P.S. — I'm realizing that there are more non-test patches that we make, such as GO_NO_VENDOR_CHECKS and GOTOOLDIR. Please treat the above list as non-exhaustive, though I'd be happy to lend eyes and hands if we'd like to make a list for documentation / investigation purposes.

[endocrimes]

GO_NO_VENDOR_CHECKS and the like should probably be disabled-by-default and opt-in tbh