build: behold the power of Nix

[maxbrunet]

Pardon the title, I couldn't help myself 😅

This PR is opened to gather opinions and is not fully ready for a release, it could be finished in this branch or with follow-up PRs (allowing others to chip in and get familiar with it). Not all required decisions have been made, none are final.

We might want to have a call to go over the changes and maybe give an intro to Nix.

I learned a whole lot about Nix by putting this PR together, and I still feel like I am only scratching the surface.

Motivation

Provide a build system and development environment where C dependencies and compilers are easy to use, without having to think about them.

Changes

• Add Nix builds for:
  • BPF program
  • Agent
  • C libraries with frame pointer and debug info
  • Docker image
• Vendor Go modules for the Nix build, otherwise a vendorSha256 hash needs to be maintained (hard to automate) (Renovate supports go mod vendor)
• Cache builds with Cachix, including intermediary builds like libraries and (cross-)compilers
• Use nixos-unstable branch of nixpkgs for latest packages (rolling release)
• Use nix flake to pin nixpkgs revision (experimental Nix code packages)
• Replace nix-shell by nix develop (part of experimental commands working with Nix flakes)
• Roughly update GitHub workflows to use Nix
• Remove Goreleaser, libbpf Git submodule, Dockerfiles
• Add containerized Nix development environment
• Update CONTRIBUTING.md

Context

Closes #1304
Closes #568
With further work, could solve #709

Pros and Cons

Cons

• New language / new paradigm for many: functional programming
• Very limited access to Git metadata (NixOS/nix#7201), no VCS data available during build (for source code reproducibility)
  • Need another tool to handle things like VERSION file and release notes (e.g. googleapis/release-please)
• Can feel invasive to install on local system for a single project (but there is a container image, and other usages are worth checking out imho: home-manager, nix-darwin, NixOS)
• Go build cache is not re-used between Nix builds, so it is slower than a barebone go build. To iterate faster during development, the later can be used in a Nix shell.

Pros

• Each build happens in its own sandbox, no configuration overlap between packages (e.g. agent, bpf program, libbpf)
• Simple cross-compilation (once you know how to 😅)
• Easy to override existing packages (e.g. eflutils, glibc, libbpf, zlib)
• Rich and up-to-date set of packages
• Reproducible builds
• Build cache usable in CI and for local development
• Has a Jsonnet-feel to it (lazy evaluation, all expressions need to return a value...)
• Strong and growing community (Discourse, Matrix, GitHub)
• NixOS Testing framework: supports virtualization and different kernels
• Integration with direnv: load Nix shell into your regular shell.

Documentation

• https://nixos.org/guides/nix-pills/
• https://nixos.org/guides/nix-pills/basics-of-language.html
• https://nixos.org/guides/nix-language.html
• https://nixos.org/manual/nixpkgs/stable/#chap-stdenv
• https://nixos.org/manual/nixpkgs/stable/#chap-cross
• https://nixos.org/manual/nixpkgs/stable/#sec-language-go
• https://nixos.org/manual/nixpkgs/stable/#sec-pkgs-dockerTools
• https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests
• https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake.html
• https://search.nixos.org/packages
• https://cachix.org

Further work

• Move more logic out of Makefiles/scripts/GH workflows into Nix
• Remove/Reduce dependence on nix develop in GH workflows
• Run Go tests from Nix
• Restructure jobs into a single workflow to maximize use of Nix cache
• Investigate pre-commit integration with Nix: cachix/pre-commit-hooks.nix

[kakkoyun]

[kakkoyun]

This looks awesome! ❤️ 💚 💙 💛 💜 Thanks for taking the time to implement this.

I need more time to wrap my head around the nix script (language?!). I have already commented on a couple of things.

I have some doubts about this approach. However, I'm open to accepting this for locally reproducible CI actions. On the one hand, I'm open to learning a new language instead of googling to do x in make every time, but on the other hand, shell scripts and makefiles are simple and familiar to everybody. A shim Makefile might be more welcoming initially.

On thing I'm sure of is we shouldn't vendor the go dependencies to the repo. Unless we need to modify the vendored packages, we shouldn't vendor. Or we shouldn't check them in the version control system. Go modules already give us reproducible builds as the go. mod file records the exact versions and commits hashes of your dependencies, which the go tool will respect and follow. vendoring bloats the repo and it makes changesets unreadable (as we can see in this PR). If this is technically needed, let's just not check them into the version control system.

https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away

[kakkoyun]

This file is used by Tilt for local development.

[maxbrunet]

Thank you for the note, I will remember to look into the integration with Tilt

[kakkoyun]

I have yet to see the end of the PR, but I wanted to note that goreleaser also handles creating the GH release and uploading the archives; I'd like to keep that automation in place.

[maxbrunet]

Yes, if we move forward with Nix, this will be covered by something else, there are still some TODOs, but it is not hard to replace.

[kakkoyun]

Awesome 🎉 It was just a reminder 😊

[maxbrunet]

A shim Makefile might be more welcoming initially.

Using Makefile like a Nix command cheat sheet? Yes, I had it in mind, definitely a good idea

On thing I'm sure of is we shouldn't vendor the go dependencies to the repo. Unless we need to modify the vendored packages, we shouldn't vendor. Or we shouldn't check them in the version control system. Go modules already give us reproducible builds as the go. mod file records the exact versions and commits hashes of your dependencies, which the go tool will respect and follow. vendoring bloats the repo and it makes changesets unreadable (as we can see in this PR). If this is technically needed, let's just not check them into the version control system.

I do not like vendoring either, but it is required by buildGoModule and Nix (with flake) will only use checked-in files for the build, the alternative is to set vendorHash/vendorSha256 and its update process is:

1. Change the current hash for a fake one in the file, otherwise the previous dependencies may be reused (so we first need to know we are going to invalidate the dependencies)
2. Run the build (wait for a while until it fails)
3. Grab the expected hash from the error output (human readable more than machine readable)
4. Update the file with the new hash

• vendorHash: is the hash of the output of the intermediate fetcher derivation. vendorHash can also take null as an input. When null is used as a value, rather than fetching the dependencies and vendoring them, we use the vendoring included within the source repo. If you’d like to not have to update this field on dependency changes, run go mod vendor in your source repo and set vendorHash = null;

https://nixos.org/manual/nixpkgs/stable/#ssec-language-go

There are discussions in nixpkgs about it, this one seems to be the lengthiest: NixOS/nixpkgs#84826

An alternative that can also be investigated is the gomod2nix community module. But it requires to maintain a gomod2nix.toml file in parallel to go.mod, which will not be covered by automation like Renovate. 😞
A very insightful read about all this: https://www.tweag.io/blog/2021-03-04-gomod2nix/

[kakkoyun]

Using Makefile like a Nix command cheat sheet? Yes, I had it in mind, definitely a good idea

❤️

I do not like vendoring either, but it is required by buildGoModule and Nix (with flake) will only use checked-in files for the build, the alternative is to set vendorHash/vendorSha256 and its update process is:

1. Change the current hash for a fake one in the file, otherwise the previous dependencies may be reused (so we first need to know we are going to invalidate the dependencies)
2. Run the build (wait for a while until it fails)
3. Grab the expected hash from the error output (human readable more than machine readable)
4. Update the file with the new hash

• vendorHash: is the hash of the output of the intermediate fetcher derivation. vendorHash can also take null as an input. When null is used as a value, rather than fetching the dependencies and vendoring them, we use the vendoring included within the source repo. If you’d like to not have to update this field on dependency changes, run go mod vendor in your source repo and set vendorHash = null;

https://nixos.org/manual/nixpkgs/stable/#ssec-language-go

There are discussions in nixpkgs about it, this one seems to be the lengthiest: NixOS/nixpkgs#84826

An alternative that can also be investigated is the gomod2nix community module. But it requires to maintain a gomod2nix.toml file in parallel to go.mod, which will not be covered by automation like Renovate. 😞 A very insightful read about all this: https://www.tweag.io/blog/2021-03-04-gomod2nix/

Thanks a lot for all the references. I'm gonna check them.

[jnsgruk]

This should be able to be significantly simplified with the recent re-base onto the Otel profiler I would imagine...

[katexochen]

the alternative is to set vendorHash/vendorSha256 and its update process is

@maxbrunet The update process isn't that bad, you can use nix-update for instance to do this in a single command (you will also need to update your vendor dir with a command, so it shouldn't differ in the amount of manual work needed, but it'll save you 1.7M lines of code). We run this script via task runner/our CI to keep the vendorHash up to date: https://github.com/edgelesssys/contrast/blob/55e6fb096a9fb3f028c93c07f21baa46473c23eb/packages/scripts.nix#L19-L50