Rule to generate docker images with nix store paths #214

jcpetruzza · 2022-04-03T13:05:18Z

Context: I'm currently using bazel with remote execution using this branch; it's not an ideal solution, but it's a first step and there could be other people interested in trying it out.

The goal was to have a docker image that contains all the nix store paths that occur in the workspace, so that the remote executors also inherit them. For this, I'm introducing a repository rule nixpkgs_docker_image, that I'm then using in my workspace as follows:

nixpkgs_docker_image(
    name = "nix_deps_image",
    srcs = [
        "@cc_toolchain_nixpkgs_info//nixpkg",
        "@nixpkgs_python_toolchain_python3//nixpkg",
        "@nixpkgs_sh_posix_config//nixpkg",
        "@rules_haskell_ghc_nixpkgs//nixpkg",
        "@nixpkgs_valgrind//nixpkg",
    ],
    bazel = "@nixpkgs_bazel//nixpkg",
    repositories = {"nixpkgs": "@nixpkgs"},
)

In srcs, one should include every usage of the nixpkgs_package rule, either explicit (as @nixpkgs_valgrind in this example) or implicit as those coming from the toolchain rules. One then gets two targets:

@nix_deps_image//:image: a genrule that uses dockerTools.buildLayeredImage to build a tarball
@nix_deps_image//:stream: a sh_binary that uses dockerTools.streamLayeredImage to output the tarball on stdout.

One then needs to ensure that the docker image is available before starting a remote build, either by manually pushing the image, or running a CI step.

All this relies on the nixpkgs_package rule creating a nixpkg subpackage in the package repository, containing the exact nix-expression used by nix-build to create the package. Ideally, this would all be more transparent to the user, who would just mention the toolchain and the package in srcs and the rule would figure out the rest, but for a proof-of-concept this is enough.

The funny bit is in the bazel argument, what's this about? There is an unexpected issue coming from using bazel provisioned by nixpkgs (which I guess is the most common case for rules_nixpkgs users): internal bazel files like @bazel_tools//tools/cpp:linux_cc_wrapper.sh.tpl get the shebang mangled so that #!/usr/bin/env bash, becomes, say #!/nix/store/5nr4gffsb5l72xpz6d5jbd0skih3r9i2-bash/bin/bash and the problem is how to ensure that those paths ends up in the docker image as well. The workaround here is for the user to optionally provide a nixpkgs_package that should correspond to the exact same bazel being used on dev, CI, etc., so that whatever was used when mangling shebangs is available as a dependency.

As far as I can see, this issue goes beyond rules_nixpkgs, in that a nixpkgs-provisioned bazel is hard to use with remote execution and may require some upstream fix. I think all the possible approaches listed in #180 would be affected by this.

This subpackage contains a default.nix that was used to build the package. The motivation is to then bundle several nixpkgs_package into a docker image

Now both types of repositories have: - A regular file called like the repository, acting as entry point. It is no longer a symlink, since it contained a user-dependent path - A ":srcs" filegroup with all sources In particular it is now possible to depend on @nixpkgs//:srcs. We want this in in order to run "nix-build" in a normal rule to generate a streamable docker image

dmadisetti · 2023-02-27T21:21:38Z

@aherrmann Regarding #291 , does this look like a reasonable approach? Wondering why this was never reviewed / commented on

aherrmann

@dmadisetti Thank you for the ping! @jcpetruzza sorry for the super late reply, this fell of the plate.

Regarding #291 , does this look like a reasonable approach?

It's targetting a different issue, namely that of providing all needed build dependencies for remote execution. It could also be used for deployables, if the deployable you're looking for are Docker images, and if you don't mind potentially an over-approximation of your runtime dependencies.

@jcpetruzza regarding the remote execution aspect. How does this approach fare related to the incrementality concerns raised in #180. I.e. if one Nix package is changed and the image needs to be rebuilt. Doesn't this invalidate all build actions and effectively force a full rebuild of the project?

cc @benradf @aloiscochard

aherrmann · 2023-03-02T10:12:14Z

core/default.nix.tpl

+  x = %{def};
+  y = if builtins.isFunction x then
+        let
+          formalArgs = builtins.functionArgs x;
+          actualArgs = builtins.intersectAttrs formalArgs {inherit %{args};};
+        in
+           x actualArgs
+       else x;
+in
+  y%{maybe_sel}


Maybe more descriptive names help make this easier to understand?

Suggested change

x = %{def};

y = if builtins.isFunction x then

let

formalArgs = builtins.functionArgs x;

actualArgs = builtins.intersectAttrs formalArgs {inherit %{args};};

in

x actualArgs

else x;

in

y%{maybe_sel}

value_or_function = %{def};

value = if builtins.isFunction value_or_function then

let

formalArgs = builtins.functionArgs value_or_function;

actualArgs = builtins.intersectAttrs formalArgs {inherit %{args};};

in

value_or_function actualArgs

else value_or_function;

in

value%{maybe_sel}

aherrmann · 2023-03-02T10:14:07Z

core/nixpkgs.bzl

+            if arg_name == None:
+                arg_name = opt
+            else:
+                arg_val = opt if arg_opt == "--arg" else "''%s''" % opt


Could this trigger quoting issues if a users passes in a quoted --argstr option?

Possibly, and now that I think about it, it may end up removing spaces in the user string. Should probably use " instead, and manually escape ", \ and ${

aherrmann · 2023-03-02T10:21:46Z

core/nixpkgs.bzl

-        nix_file = cp(repository_ctx, repository_ctx.attr.nix_file)
-        expr_args = [repository_ctx.path(nix_file)]
+        src = repository_ctx.attr.nix_file
+        cp(repository_ctx, src, label_to_path(src, prefix="nixpkg/src"))


Could you explain what this change does and why it's needed? IIUC this introduces an extra prefix and nests the copied files deeper. Wouldn't that break cross workspace location expansion references?

It's been a while, but I think the idea was that we want to have a nixpkgs/default.nix on every package, but then we need to prevent this interfering with user provided files, so by putting the user input under nixpkgs/src, we ensure there will be no conflict

Hairiest merge res is around here. Didn't dig deep enough to figure out how to resolve- just took changes and hoped it worked

I see, rules_nixpkgs already creates a special sub-directory called bazel-support, which houses the Nix output link used as GC-root.

So, user-code already has to avoid collision with that reserved name.

Would it be possible to generate the nixpkgs/default.nix under that sub-directory instead? Then this change wouldn't need to introduce the nixpkgs/src prefix.

jcpetruzza · 2023-03-03T00:20:30Z

Hi @aherrmann!

regarding the remote execution aspect. How does this approach fare related to the incrementality concerns raised in #180. I.e. if one Nix package is changed and the image needs to be rebuilt. Doesn't this invalidate all build actions and effectively force a full rebuild of the project?

Good question; I guess it depends on whether bazel considers the container-image of a (host) platform definition part of the build input or not, but I don't really know what it does here. My setup follows this guide, so essentially I have something like this:

_NIX_DEPS_IMAGE_TAG = "f3279413...."

platform(
    name = "nix_deps_image_platform",
    constraint_values = [
        "@bazel_tools//platforms:x86_64",
        "@bazel_tools//platforms:linux",
        "@bazel_tools//tools/cpp:clang",
    ],
    exec_properties = {
        "OSFamily": "Linux",
        "container-image": "docker://some.registry/nix-deps-image:%s" % _NIX_DEPS_IMAGE_TAG,
    },
    parents = [
        "@io_tweag_rules_nixpkgs//nixpkgs/platforms:host",
    ],
)

And to use RBE I then do something like:

bazel build --remote_executor=grpcs://remote.buildbuddy.io --host_platform=//:nix_deps_image_platform //...

So every time I bump/add a nix dependency, I push a new version of the image and update the value of _NIX_DEPS_IMAGE_TAG. That's the only change from the pov of bazel; as I said, I don't know what it does here (I mostly update the image when I bump the version of nixpkgs, and expect everything to be rebuilt anyway, so didn't pay attention)

dmadisetti

Cool!

dmadisetti · 2023-03-03T14:45:03Z

core/nixpkgs.bzl

-        nix_file = cp(repository_ctx, repository_ctx.attr.nix_file)
-        expr_args = [repository_ctx.path(nix_file)]
+        src = repository_ctx.attr.nix_file
+        cp(repository_ctx, src, label_to_path(src, prefix="nixpkg/src"))


Hairiest merge res is around here. Didn't dig deep enough to figure out how to resolve- just took changes and hoped it worked

dmadisetti · 2023-03-03T14:46:04Z

experimental/docker.bzl

+
+    contents = []
+    for src in srcs:
+        path_to_default_nix = repository_ctx.path(src.relative("default.nix"))


Default nix seems to resolve to the consuming library, is this correct? Maybe we should document that a default.nix is required

touch default.nix seemed to let things run for me. But should be more general (e.g. flakes, or nothing)

Yeah, seems to want all deps to have a default.nix, even non-nix sources

Error in path: Not a regular file: /home/user/.cache/bazel/_bazel_user/8bbdcf70ee87482f767b06aac75f66d1/external/com_google_absl/absl/flags/default.nix

Could just check? I'll look into this a bit more

dmadisetti · 2023-03-03T14:46:56Z

experimental/docker/default.nix.template

+    inherit name tag;
+
+    contents = [
+      # Contents get copied to the top-level of the image, so we jus putt


aherrmann · 2023-03-06T09:09:22Z

I guess it depends on whether bazel considers the container-image of a (host) platform definition part of the build input or not, but I don't really know what it does here.

AFAIK it does, see here. So, I would expect this to invalidate the entire build. That also matches what we've seen in experiments in the past.

I mostly update the image when I bump the version of nixpkgs, and expect everything to be rebuilt anyway, so didn't pay attention

Fair enough. How problematic this is depends on the use-case. Indeed, bumping the nixpkgs revision will likely invalidate most targets in a project, but is also not expected to happen very frequently. Adding or tweaking individual packages, e.g. Nix imports of utilities developed separately, or third-party dependencies, OTOH should ideally not have that problem.

… paths)

benradf · 2023-07-17T09:57:06Z

Closing this as it has been superseded by #351. Will try and get that PR merged instead.

jcpetruzza added 3 commits April 2, 2022 17:28

Add a "nixpkg" package to every nixpkgs_package repo

d1c0f75

This subpackage contains a default.nix that was used to build the package. The motivation is to then bundle several nixpkgs_package into a docker image

Add rule to generate docker images from packages

71296d3

aherrmann reviewed Mar 2, 2023

View reviewed changes

dmadisetti reviewed Mar 3, 2023

View reviewed changes

dmadisetti added a commit to cemel-jhu/rules_nixpkgs that referenced this pull request Mar 15, 2023

Variation of tweag#214 (Rule to generate docker images with nix store…

c41ae45

… paths)

dmadisetti mentioned this pull request Mar 15, 2023

Variation of #214 (Rule to generate docker images with nix store paths) #351

Open

3 tasks

benradf closed this Jul 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rule to generate docker images with nix store paths #214

Rule to generate docker images with nix store paths #214

jcpetruzza commented Apr 3, 2022

dmadisetti commented Feb 27, 2023

aherrmann left a comment

aherrmann Mar 2, 2023

aherrmann Mar 2, 2023

jcpetruzza Mar 3, 2023

aherrmann Mar 2, 2023

jcpetruzza Mar 3, 2023

dmadisetti Mar 3, 2023

aherrmann Mar 6, 2023

jcpetruzza commented Mar 3, 2023

dmadisetti left a comment

dmadisetti Mar 3, 2023

dmadisetti Mar 3, 2023

dmadisetti Mar 3, 2023

dmadisetti Mar 3, 2023

dmadisetti Mar 3, 2023

aherrmann commented Mar 6, 2023

benradf commented Jul 17, 2023

Rule to generate docker images with nix store paths #214

Rule to generate docker images with nix store paths #214

Conversation

jcpetruzza commented Apr 3, 2022

dmadisetti commented Feb 27, 2023

aherrmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcpetruzza commented Mar 3, 2023

dmadisetti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aherrmann commented Mar 6, 2023

benradf commented Jul 17, 2023