[RFC 140] Simple package paths, part 1b: Enabling the directory structure

[infinisil]

Note
This was reviewed by and merged on 2023-09-05 15:30 CEST with the Nixpkgs Architecture Team and others that joined the call!

Context

This is the first part of the implementation of the accepted RFC 140, the introduction of the new directory structure. Note that this was split up into Part 1a (#250885) and Part 1b (this PR). The second part will be done afterwards, it's draft can be found here: #211832. This is hopefully only the first of many quality-of-life improvements to Nixpkgs by the Nixpkgs Architecture Team.

Overview

This RFC introduces the new pkgs/by-name directory, which makes it possible to add a new top-level package by just declaring it in pkgs/by-name/${prefix}/${name}/package.nix, where name is the attribute name the package should be exposed as, and prefix is the lowercased two-letter prefix of the attribute name. This means that for a lot of packages there's no more need to find an appropriate location for the file in the loosely categorized pkgs/ hierarchy, and no more editing of the all-packages.nix file!

Restrictions

Note however that only a subset of packages may be using this new directory structure for now, extending this is future work for the Nixpkgs Architecture Team. In particular these requirements have to be met for an attribute to be defined in pkgs/by-name:

• It's at the top level, not allowed are packages from package sets like pkgs.python3Packages.requests.
• It's a derivation, not allowed are non-derivations like pkgs.fetchFromGitHub.
• It's defined using callPackage, not allowed are packages defined using functions like python3Packages.buildPythonApplication.

Things done

• Implement the automatic calling of packages in the pkgs/by-name directory and move pkgs.hello to it
  • Working
  • Optimised
  • Tested
  • Clean and documented code
• Implement CI checks enforcing the structure and semantics of the pkgs/by-name directory
  • Working, you can run nix-build -A tests.byName.nixpkgs
  • Is getting run by CI (see the checks in this PR)
    • Invalid pkgs/by-name breaks CI (example)
  • Tested, you can run nix-build -A tests.byName.tests
  • Clean and documented code
• Update the contributor documentation for the pkgs/by-name directory and recommend using it for new packages
• Add myself to relevant code owners entries

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-13-nixpkgs-architecture-team-meeting-40/29087/1

[gilice]

I am not sure you knew you can interpolate format strings in rust :)

Left diffs if you want to do it this way, but because it's mostly stylistic, I won't push it if you don't like it

[infinisil]

@gilice I did not know that, neat! Greatly appreciated, I barely know Rust :D

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-13-documentation-team-meeting-notes-63/30450/1

[oxij]

Related NixOS/rfcs#109 (comment).

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-05-nixpkgs-architecture-team-meeting-43/32635/1

Everything was addressed

[infinisil]

As planned we looked at this together in today's meeting with the Nixpkgs Architecture Team (including @aakropotkin, @roberth, @DavHau, @phaer), and @fgaz, @quantenzitrone and @YorikSar.

Everything was looked at and minor documentation updates were done. Nobody had any complaints about this PR, so we'll go ahead with merging it! 🚀

[github-actions]

Backport failed for release-23.05, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git fetch origin release-23.05
git worktree add -d .worktree/backport-237439-to-release-23.05 origin/release-23.05
cd .worktree/backport-237439-to-release-23.05
git checkout -b backport-237439-to-release-23.05
ancref=$(git merge-base f67a4c3f12a72f474a4349f6255f4a0b24b805ff 77d50b03e4388f22e1f36a2621a9287a12a138be)
git cherry-pick -x $ancref..77d50b03e4388f22e1f36a2621a9287a12a138be

[infinisil]

Backport PR, let's make sure this one doesn't have any problems before merging it though: #253442

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-init-generate-nix-packages-from-urls-with-hash-prefetching-dependency-inference-license-detection-and-more/25035/18

[doronbehar]

I'm getting the following error from CI on a new PR I opened, from the check-by-name.yml workflow:

  Error: fatal: couldn't find remote ref refs/pull/255902/merge
  The process '/usr/bin/git' failed with exit code 128

[infinisil]

Thanks for the report! Opened #256756 to discuss this

[SuperSandro2000]

IMO this should just be default.nix to follow the previous pattern. The argument for naming it package.nix is rather weak and just creates confusion since there is no real reason to name it different.

[infinisil]

This is part of the RFC so we can't just change that, and you already started a discussion on this in NixOS/rfcs#140 (comment), let's keep it in one place

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/ci-will-soon-enforce-pkgs-by-name-for-new-packages/38098/1