Shipping Security Updates

[nbp]

Motivations

The way of handling security updates today involves a lot of humans efforts, going from reading the mailing list, to adding lines to the configuration files.

More over, this implies having control over the top-level derivation, which is not always possible with nix-env, and nix-shell. This has to be patched inside the current pipeline, as what got made for NixOS.

Also, as the current method for applying security updates is complex, we are only restricting them to a few packages, such as glibc, bash and openssl which are used a lot, but we do not such mechanism for all other packages which have minor security / usability updates such as firefox.

As a work-around, today, we offer the option of having a small channel, to get updates from, but this is not a good solution for end-users, as the small channel basically involves recompiling any large project.

More over the current mechanism cannot safely be composed as explained in #4336.

Goals

This work aims at making it transparent, such as security updates are cannot be distinguished from any other updates by the end user.

This work aims at making it easy, such that packagers only have to cherry-pick a commit made on master into a security/master branch to let the buildfarm compile and distribute the package against the latest stable branch.

This work aims at making it fast, such that the buildfarm in charge of producing pre-compiled versions of the packages will only recompile the packages which are being fixed, against the latest stable branch. When a user updates, Nix will download from the latest stable channel, with the addition of the recompiled packages, and Nix will patch all the packages depending on the fixed packages.

This new approach is sane, as it constructs the patches which have to be apply, without any knowledge of the runtime dependency. Thus, all the packages patches can be build up-front without having to realize any of the derivations. This is a big plus to guarantee reproducible builds of the patching mechanism.

Approach

Nixpkgs as a pure function

This approach aims at constructing patches based on the Nix expressions of the packages. Thus we have to change the way we evaluate Nixpkgs to reflect the patching mechanism.

This approach is based on the recent modification of #14000 (#9400), which make Nixpkgs behave as a single function with no external inputs. As the Nix language is pure, the following property holds for Nixpkgs.

    fix f == f (fix f)

This property means that we can call f one extra time on the result of the fix-point without changing the result of the fix-point. As a consequence, all the packages will keep the same hashes.

Recompile only fixed packages

This patching mechanism relies on one hypothesis. This hypothesis is that all packages derivations are expressed in the last evaluation of Nixpkgs, and that dependencies are taken from the previous evaluation of Nixpkgs.

This hypothesis does not hold, as of today, but with the help of a static analysis, we can identify how many times we jump through the Nixpkgs argument before we settle on a derivation, and before we settle on its inputs.

Once this hypothesis is guaranteed, we can change the above expression to

    g (fix f)

Where g is a slightly different version of f. With the above hypothesis, this means that all derivations of packages are taken from g, but the dependencies of g are taken from f.

If f is the stable version of Nixpkgs, and g is a stable version of Nixpkgs with extra patches. This implies we will recompile any patched packages of g, while keeping the same hashes as f for non-patched packages.

At this stage, we have what we expect to let Hydra compile. Still, this stage does not apply any patches to any of the packages.

Building patches

To be able to apply patches to the dependencies, we need the equivalent of a fix-point, to evaluate the dependencies, and the dependencies of the dependencies, and so on … In addition to a fix-point, we want to iterate from the maybe-recompiled packages of g (fix f), and iterate over g to settle on the patched version of the packages. We would name this patched version of packages h.

To construct patches, we need to know the dependencies of the packages, and which one got updated.

To list the dependencies of a package, we have to rely on some form of introspection, encoded as part of the derivations. As a first approach I relied on callPackage, which would give us a super set of the dependencies, but this has a lot of issues, and left-over packages. As a second approach, I now rely on the buildInputs which are exposed by the derivation, but this has the issue of being a sub-set of the dependencies. Still, I am confident that we can instrument the default builder to ensure that buildInputs are a super set of the runtime dependencies.

To check for updates, we can compare the outPath of 2 derivations. If the derivations have a different outPath, this means that the derivations has a different inputs. (Note: hash collisions are unlikely, but if this happens we can easily add a dummy environment variable to get a different hash)

Patches are composed of the hashes of the old versions, and the hashes of the new versions. Making a patch for one derivation, is being able to list the old hashes and associate them with the new hashes. The hashes are the hashes of the dependencies.

Based on the last hypothesis, looking for the dependencies of g (fix f), will gives us hashes of derivations from fix f. These are our old hashes.

Looking for the new hashes, implies that we have to build packages based on patched versions of the packages, i-e h. Thus if we look for the dependencies of g h, we would have our new hashes.

Therefore h definition should look like:

  h = patch (g (fix f)) (g h);

Where patch is a function which on a derivation, will take the value of g (fix f), and apply a patch for any of its dependencies which got updated, by substituting the dependencies from fix f by dependencies from h.

In practice, we also need to feed fix f to this patch function, as we have to make sure that patches can be applied safely, by ensuring that the outPath of the derivation still have the same length.

Status

• Implement this approach on top of Nixpkgs
• Add test case to ensure that, when the one-application hypothesis holds, we can only recompile fixed packages, and apply patches on the rest of the packages.
• Make a static analysis which reports all violations of the one-application hypothesis.
• nix-env -f ./. -qaP --drv-path works when quickfixPackages is equal to defaultPackages, and does not cause recompilation of existing packages. (Note: nox-review fails to see that we have no recompilation because of violation of the one-application hypothesis)
• Add a test case to verify that nix-env -f ./. -qaP --drv-path always works, when quickfixPackages is equal to defaultPackages.
• Hook the test cases as release blockers of Nixpkgs.

This mechanism is disabled by default, as no quickfixPackages are provided yet. Also, when disabled, this work has no impact.

To avoid to bloat and delay these changes, I would recommend that we merge them as soon as possible, even if the one-application hypothesis does not hold yet, and address any issues reported by the static analysis as follow-up issues.

References

NixCon slides: http://nbp.github.io/slides/NixCon/2015.ShippingSecurityUpdates/
NixCon video: https://www.youtube.com/watch?v=RhcKXS00zEE

[PyroLagus]

Thank you! This is much needed. I'm excited to see how this will turn out :)

[nbp]

@Mathnerd314 :

There's some existing work in https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/replace-dependency.nix if you haven't seen it.

I saw it, and this is exactly why I am not using it, and why I am making this work. I will explain more next Sunday in my talk.

@Mathnerd314 :

I guess this is a hack until the promised "Optimize this function by only using runtime dependencies of the original package set" is implemented?

No, in fact, this is exactly the opposite. The hack is to rely on runtime dependencies, as runtime dependencies are not known statically (without building packages). The current implementation is a conservative approximation of runtime dependencies. There is still one issue with the current function, as it does not consider the closure of dependencies, but only explicit dependencies.

[nbp]

This extra patches are changing the design to fit better with the ideal model shown in the slide of my presentation. [1][2]

So far, while evaluating packages with nix-env -f ./. -qaP --drv-path I saw multiple issues.

1. Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.
2. So far, the code is unable to retrieve the list of dependencies of 3190 packages, which means that for these packages we would not patch them if one of their dependencies got modified.

[1] http://nbp.github.io/slides/NixCon/2015.ShippingSecurityUpdates/
[2] https://www.youtube.com/watch?v=RhcKXS00zEE&list=PL_IxoDz1Nq2Y7mIxMZ28mVtjRbbnlVdmy&index=13

[Ericson2314]

Uh, no it is. :)

[Ericson2314]

It won't render all-packages.nix for me to do a line comment, but the documentation the of self (the parameter) says self is affected by user overrides, but since it is fixed within pkgFun I don't think it is.

[Ericson2314]

lib is let-bound in all-packages.nix, but since it is also inherited in the body of the function this could be made more concise by doing just lib = import ../lib;. Since with pkgs; comes after the let-binding of lib, either way were a user to override lib that would take priority.

[Ericson2314]

@nbp I've only glanced at your slides, so can't yet speak to that. But even in terms of the refactoring alone, this is a HUGE improvement. Thanks so much! (and @Mathnerd314 too!)

[Ericson2314]

Oh and @Mathnerd314 got a good portion of my comments too, hah, but they didn't show up on the diff.

[Ericson2314]

nit: 4 space indent

[Mathnerd314]

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.

This is because nix-env uses pointer equality for skipping duplicate derivations, rather than output path or derivation path: https://github.com/NixOS/nix/blob/master/src/libexpr/get-drvs.cc#L193. Personally, I would remove that check; outputting all possible aliases is consistent with the behavior of command-not-found.

[Mathnerd314]

I rebased the patches; still testing it.

[nbp]

from @Ericson2314

If we remember the hashes of things we are going to rebuild, and we find something we wouldn't rebuild that has the same hash as something we are rebuilding, we could "rebuild" it anyways. That will work for simply rebound aliases, but not for alises that involve overriding (an example of introduction >= elimination, but introduction > 0).

I intended to do something similar to ensure that we do not add such aliases to nixpkgs.

I do not think this is a good choice to do this automatically as this effectively means that for installing any sotfware you will have to evaluate all of them.

from @Mathnerd314

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.

This is because nix-env uses pointer equality for skipping duplicate derivations, rather than output path or derivation path: https://github.com/NixOS/nix/blob/master/src/libexpr/get-drvs.cc#L193. Personally, I would remove that check; outputting all possible aliases is consistent with the behavior of command-not-found.

That's what I thought at first, but then I realized that these differences are related to aliased packages, and that for each newly reported entry, we have a potential security issue (as described in #10851 (comment))

So even if this behaviour of nix-env is not consistent, we should still fix these issues.

[Ericson2314]

@nbp Sorry for taking a while to respond, but I think that all makes sense now. Thanks for explaining it.

I wonder if we should revive #10307 after all, as a diff this big goes stale so quick.

[Ericson2314]

Also Ericson2314@03486c8 here is a commit just with the improved documentation for overridePackages you might want to add to this.

[jb55]

I got codetriaged here. What's the status of this?

[Ericson2314]

I'm not sure about the status of the main security part, but the improved fix-point part has been good to go for a while, hense my advocacy to split this in two.

[PyroLagus]

Wait, so if the improved fix-point is already finished, what do we have to do to make it work and patch security updates? I'd really like to help on this, but I have no idea how.

[nbp]

@spacekitteh ,

what sort of fixes are holding up a working demo?

There is no mandatory fixes but this would be a nicer experience to demo a larger set of fixable packages.

From the infrastructure, the main blocker for getting a working demo is to get the security-updates branch built in Hydra, i-e making rules to build the base branch and merge it with the security branch.

Then, we need some task-force to get some fixes (or dummy fixes) applied in the security updates branch.

The static analysis is only a tool made to highlight the parts which would not work properly. I have not yet measured, but I would expect that we have already have a large number of packages with no issues. Among the prerequisite are that I should write some tests to ensure that we can successfully update real packages such as openssl.

Anything that actually breaks the mechanism?

The mechanism can already be tested locally. For that you need to check out the security-updates branch twice, in 2 different directories, and call the default.nix file from one with a quickfixPackages argument which imports the pkgs/top-level/index.nix file of the other. The other directory contains all the program which have security patches.

Currently, we could merge the security-updates branch as-is, as the patching mechanism is disabled unless the quickfixPackages argument is provided.

Or is it that you're trying to fix all of nixpkgs of static analysis errors?

That is something that I would hardly be able to complete alone. Writing documentation to explain how to fix them would be hard as people tend to innovate every-time they write new packages.

One of the restriction of the security-updates branch is that all packages should use callPackage, and that callPackage should provide all the input of the package. We should not use callPackages, as this suggest that people can use rec to get other packages, instead of getting them from the arguments of callPackage.

[infinisil]

Closing because this seems to be stalled (for 2 years now), feel free to reopen when work is being done again.

[nixos-discourse]

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/in-overlays-when-to-use-self-vs-super/2968/5

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-10-19-nixpkgs-architecture-team-meeting-14/22578/3

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/timely-updates-for-nixos/29231/7

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/correct-way-to-export-functions-in-overlays-and-use-them-in-others/45016/2