nixos/nix-daemon: use structural settings #139075

polykernel · 2021-09-23T03:24:52Z

The nix.* options, apart from options for setting up the
daemon itself, currently provide a lot of setting mappings
for the Nix daemon configuration. The scope of the mapping yields
convience, but the line where an option is considered essential
is blurry. For instance, the extra-sandbox-paths mapping is
provided without its primary consumer, and the corresponding
sandbox-paths option is also not mapped.

This increases the maintenance burden as maintainers have to
closely follow upstream changes which in the case of Nix is made
worse by having to juggle between two state versions, the stable release
and the unstable release, which may include or deprecate features.

One conditional option publicHostKey was added to the buildMachines
submodule corresponding to the newly added base64 encoded public host
key setting exposed in the builder syntax. The build machine generation
was subsequently rewritten to use concatStringsSep to group concatenations.

This commit aims to following the standard outlined in RFC 42[1] to
implement a structural setting pattern. The Nix configuration is encoded
at its core as key-value pairs which maps nicely to attribute sets, making
it feasible to express in the Nix language itself. Some existing options are
kept such as buildMachines and registry which present a simplified interface
to managing the respective settings.

Legacy configurations are mapped to their corresponding options under the
structural settings for backwards compatibility.

Various options settings in other nixos modules and relevant tests have been
updated to use the structural setting for consistency.

The generation and validation of the configration file has been modified to
use writeTextFile instead of runCommand for clarity. Note that validation
is now mandatory as strict checking of options has been pushed down to the
derivation level due to freeformType consuming unmatched options. Furthermore,
validation can not occur when cross-compiling due to current limitations.

[1] - https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md

Motivation for this change

Initial effort for nix-community/home-manager#2324

Things done

nixos-discourse · 2021-10-21T08:20:04Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/local-flake-based-nix-search-nix-run-and-nix-shell/13433/12

nixos/modules/system/boot/binfmt.nix

roberth · 2021-10-21T12:17:37Z

nixos/modules/services/misc/nix-daemon.nix

+
+      settings = mkOption {
+        type = types.submodule {
+          freeformType = semanticConfType;


Having a freeformType means that we don't have implicit checking for typos anymore.
Can we check the config file in a derivation using nix.package?
Otherwise this is somewhat of a regression. For some modules this would be an acceptable trade-off, but for such a core element of NixOS it is not.
Perhaps using something like

HOME=..... nix show-config |& sed -e 's/^warning:/error:/' | (! grep unknown\ setting)

You are right, using freeformType causes typos to remain silent. I think we can incorporate checking into the checkPhase of the derivation which generates nix.conf, but this has the side effects that validation for options specified in the submodule do not occur at the module system level. This essentially means typos and invalid settings are now indistinguishable. This is ok but checking will always have to occur instead of being optional.

An alternate approach is to push the freeformType one level down, so nix.settings becomes a standard submodule, and a descendant typed option like nix.settings.extraConfig is used to for additional settings. This also means merging behavior of the the extra settings and the "main" settings have to be handled manually and outside of the module system.

I think the first approach(your suggestion) is easier to implement and less error prone, but I am not sure if having the error/warning at the derivation level as opposed to the module system level is acceptable.

but I am not sure if having the error/warning at the derivation level as opposed to the module system level is acceptable.

I'm still a novice here. IMHO, the modular configurations eventually comes down to the derivation that generates the nix.conf file. As the generation process itself is already outside (or underneath) the modular expressions, the vlidation of the output file nix.conf would not create an extra "non-modulalized" part.

By the way, there are some modules who use the generator attribute provided by type to generate the output file, where the type attribute is specified under the options tree. Maybe the validation could be specified as part of the type or options to go even more "modular". This would probably go through some "consesus-reaching" process such as an additional issue or an RFC, and modules using a rather imperative way of validation would then be modified to fit into the standardized pattern.

By the way, there are some modules who use the generator attribute provided by type to generate the output file, where the type attribute is specified under the options tree. Maybe the validation could be specified as part of the type or options to go even more "modular".

Could you elaborate on the validation mechanism, I am not sure I understand correctly but do you mean using lib.generators to generate the config which itself and specifying a schema under an option which gets match to its value? The issue when I thought about it is using a schema which is analogous to the mapping all config options in the options tree is it has to be total and essentially match upstream, which increases maintenance burden and is what RFC 42 tries to avoid.

I cannot recall where I saw them. Maybe I have mixed something up. Sorry for that.

It seems that the generators produce strings instead of files. It then makes more sense to attach the validation through package overriding or similar approaches.

@polykernel

not sure if having the error/warning at the derivation level as opposed to the module system level is acceptable.

While earlier is better, it is still early enough, as derivation builds are also free of side effects.

@ShamrockLee You're thinking of pkgs.formats. Not being in lib, those can produce a derivation. It's also a better name :)

Maybe the validation could be specified as part of the type or options to go even more "modular". This would probably go through some "consesus-reaching" process such as an additional issue or an RFC, and modules using a rather imperative way of validation would then be modified to fit into the standardized pattern.

The separation we have right now, with formats producing a type that returns plain Nix values instead of a derivation, is quite useful because it allows modules to use values from the config. This wouldn't be possible if the type converted the settings to a derivation directly.
I suppose we could use the formats as 'circular programs' though, taking more advantage of laziness. See #144216

dasJ

Big 👍 for making the settings consistent with the nix manual and for using a much nicer way of specifying them. However I am not sure if this should go into the 21.11 release or if we should wait for 22.05. In any case we should probably wait for the branch-off and backport to 21.11 if necessary

ShamrockLee · 2021-12-22T00:25:16Z

It has been a while after the 21.11 branch-off.

Looking forward to its merging!

thiagokokada · 2022-01-20T20:53:05Z

Is there something blocking this merge?

(Except from the conflicts).

roberth · 2022-01-21T12:46:58Z

I don't see any blockers and the conflict seems simple.

polykernel · 2022-01-26T04:08:51Z

I have resolved the conflict and updated the publicHostKey option to be static.

ShamrockLee · 2022-01-26T11:11:06Z

As the target changelog release note is 21.11, this is to be back-ported to release-21.11. Is that right?

It seems alright as the backward-compatibility is kept with the mapping layer.

dasJ · 2022-01-26T11:20:31Z

Thanks for pointing this out, I was close to merging this ;)
I don't think we should backport this since it introduces a lot of warnings for existing setups and the large size of this PR leaves room for regressions.
If the release notes are moved to 22.05, I will happily merge this (unless @roberth or anyone else intervenes)

roberth

Small things and off-topic things. Looking pretty good otherwise.

nixos/modules/services/continuous-integration/hydra/default.nix

roberth · 2022-01-26T13:06:31Z

nixos/tests/os-prober.nix

+    nix.settings = {
+      substituters = lib.mkForce [];
+      hashed-mirrors = null;
+      connect-timeout = 1;
+    };


There's a bunch of these. Should probably be the default for VM tests. Not a requirement for this PR though.

nixos/modules/services/misc/nix-daemon.nix

roberth · 2022-01-26T13:20:01Z

nixos/modules/services/misc/nix-daemon.nix

    assertions =
-      let badMachine = m: m.system == null && m.systems == [];
-      in [
+      let badMachine = m: m.system == null && m.systems == [ ];
+      in
+      [
        {
-          assertion = !(builtins.any badMachine cfg.buildMachines);
+          assertion = !(any badMachine cfg.buildMachines);
          message = ''
            At least one system type (via <varname>system</varname> or
              <varname>systems</varname>) must be set for every build machine.
              Invalid machine specifications:
          '' + "      " +
-          (builtins.concatStringsSep "\n      "
-            (builtins.map (m: m.hostName)
-              (builtins.filter (badMachine) cfg.buildMachines)));
+          (concatStringsSep "\n      "
+            (map (m: m.hostName)
+              (filter (badMachine) cfg.buildMachines)));
        }
      ];


This would benefit from a second attempt at submodule assertions, #97023 (comment), both in code complexity and error message quality.
(off-topic)

nixos/doc/manual/from_md/release-notes/rl-2111.section.xml

The `nix.*` options, apart from options for setting up the daemon itself, currently provide a lot of setting mappings for the Nix daemon configuration. The scope of the mapping yields convience, but the line where an option is considered essential is blurry. For instance, the `extra-sandbox-paths` mapping is provided without its primary consumer, and the corresponding `sandbox-paths` option is also not mapped. The current system increases the maintenance burden as maintainers have to closely follow upstream changes. In this case, there are two state versions of Nix which have to be maintained collectively, with different options avaliable. This commit aims to following the standard outlined in RFC 42[1] to implement a structural setting pattern. The Nix configuration is encoded at its core as key-value pairs which maps nicely to attribute sets, making it feasible to express in the Nix language itself. Some existing options are kept such as `buildMachines` and `registry` which present a simplified interface to managing the respective settings. The interface is exposed as `nix.settings`. Legacy configurations are mapped to their corresponding options under `nix.settings` for backwards compatibility. Various options settings in other nixos modules and relevant tests have been updated to use structural setting for consistency. The generation and validation of the configration file has been modified to use `writeTextFile` instead of `runCommand` for clarity. Note that validation is now mandatory as strict checking of options has been pushed down to the derivation level due to freeformType consuming unmatched options. Furthermore, validation can not occur when cross-compiling due to current limitations. A new option `publicHostKey` was added to the `buildMachines` submodule corresponding to the base64 encoded public host key settings exposed in the builder syntax. The build machine generation was subsequently rewritten to use `concatStringsSep` for better performance by grouping concatenations. [1] - https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment)

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see #139075 (comment) (cherry picked from commit 2d0f4a7)

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment) backports: 2d0f4a7

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment)

polykernel marked this pull request as draft September 23, 2021 03:25

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Sep 23, 2021

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Sep 23, 2021

github-actions bot added 8.has: changelog 8.has: documentation labels Sep 23, 2021

ofborg bot added the ofborg-internal-error Ofborg encountered an error label Oct 10, 2021

polykernel marked this pull request as ready for review October 10, 2021 16:59

polykernel requested a review from joachifm as a code owner October 10, 2021 16:59

ofborg bot requested a review from roberth October 10, 2021 17:22

ambroisie reviewed Oct 21, 2021

View reviewed changes

nixos/modules/system/boot/binfmt.nix Outdated Show resolved Hide resolved

roberth reviewed Oct 21, 2021

View reviewed changes

cole-h removed the ofborg-internal-error Ofborg encountered an error label Oct 21, 2021

roberth mentioned this pull request Nov 2, 2021

Simplify RFC42 settings declarations using circular programs #144216

Open

ofborg bot requested a review from roberth November 3, 2021 02:37

polykernel requested a review from dasJ as a code owner November 19, 2021 02:40

dasJ approved these changes Nov 20, 2021

View reviewed changes

ShamrockLee mentioned this pull request Jan 4, 2022

RFC42 tracking issue #144575

Open

berbiche mentioned this pull request Jan 20, 2022

Module request: Port the nix modules from NixOS nix-community/home-manager#2324

Closed

aanderse mentioned this pull request Jan 25, 2022

nix: allow to define nix.extraConfig as a set #156682

Closed

13 tasks

roberth requested changes Jan 26, 2022

View reviewed changes

roberth added a commit to hercules-ci/nixpkgs that referenced this pull request Dec 1, 2023

nixos/nix.nix: Support new Nix 2.20 command syntax

2d0f4a7

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment)

roberth mentioned this pull request Dec 1, 2023

nix-config.drv fails to build with nix from master #271146

Closed

github-actions bot pushed a commit that referenced this pull request Dec 8, 2023

nixos/nix.nix: Support new Nix 2.20 command syntax

31b97b2

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see #139075 (comment) (cherry picked from commit 2d0f4a7)

github-actions bot pushed a commit that referenced this pull request Dec 8, 2023

nixos/nix.nix: Support new Nix 2.20 command syntax

beb1e24

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see #139075 (comment) (cherry picked from commit 2d0f4a7)

bacchanalia added a commit to awakesecurity/nixpkgs that referenced this pull request May 7, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

6f63e1e

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment) backports: 2d0f4a7

bacchanalia mentioned this pull request May 7, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax awakesecurity/nixpkgs#221

Merged

bacchanalia added a commit to awakesecurity/nixpkgs that referenced this pull request May 7, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

29464f3

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment) backports: 2d0f4a7

tm-drtina pushed a commit to awakesecurity/nixpkgs that referenced this pull request May 16, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

7554c31

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment) backports: 2d0f4a7

tm-drtina pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jun 7, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

61b46d9

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment) backports: 2d0f4a7

tm-drtina pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 4, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

9ec155e

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment) backports: 2d0f4a7

tm-drtina pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 22, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

10ff95b

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment)

tm-drtina pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 29, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

cfd2529

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment)

tm-drtina pushed a commit to awakesecurity/nixpkgs that referenced this pull request Aug 13, 2024

nixos/nix.nix: Support new Nix 2.20 command syntax

f63db86

Otherwise it shows a deprecation warning, which is escalated to an error. For context, see NixOS#139075 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/nix-daemon: use structural settings #139075

nixos/nix-daemon: use structural settings #139075

polykernel commented Sep 23, 2021 •

edited

Loading

nixos-discourse commented Oct 21, 2021

roberth Oct 21, 2021

polykernel Oct 22, 2021 •

edited

Loading

polykernel Oct 22, 2021

ShamrockLee Oct 31, 2021 •

edited

Loading

polykernel Nov 1, 2021

ShamrockLee Nov 2, 2021 •

edited

Loading

roberth Nov 2, 2021

dasJ left a comment

ShamrockLee commented Dec 22, 2021

thiagokokada commented Jan 20, 2022

roberth commented Jan 21, 2022

polykernel commented Jan 26, 2022

ShamrockLee commented Jan 26, 2022

dasJ commented Jan 26, 2022

roberth left a comment

roberth Jan 26, 2022

roberth Jan 26, 2022

nixos/nix-daemon: use structural settings #139075

nixos/nix-daemon: use structural settings #139075

Conversation

polykernel commented Sep 23, 2021 • edited Loading

Motivation for this change

Things done

nixos-discourse commented Oct 21, 2021

roberth Oct 21, 2021

Choose a reason for hiding this comment

polykernel Oct 22, 2021 • edited Loading

Choose a reason for hiding this comment

polykernel Oct 22, 2021

Choose a reason for hiding this comment

ShamrockLee Oct 31, 2021 • edited Loading

Choose a reason for hiding this comment

polykernel Nov 1, 2021

Choose a reason for hiding this comment

ShamrockLee Nov 2, 2021 • edited Loading

Choose a reason for hiding this comment

roberth Nov 2, 2021

Choose a reason for hiding this comment

dasJ left a comment

Choose a reason for hiding this comment

ShamrockLee commented Dec 22, 2021

thiagokokada commented Jan 20, 2022

roberth commented Jan 21, 2022

polykernel commented Jan 26, 2022

ShamrockLee commented Jan 26, 2022

dasJ commented Jan 26, 2022

roberth left a comment

Choose a reason for hiding this comment

roberth Jan 26, 2022

Choose a reason for hiding this comment

roberth Jan 26, 2022

Choose a reason for hiding this comment

polykernel commented Sep 23, 2021 •

edited

Loading

polykernel Oct 22, 2021 •

edited

Loading

ShamrockLee Oct 31, 2021 •

edited

Loading

ShamrockLee Nov 2, 2021 •

edited

Loading