allow installables without drvPath #7417

tomberek · 2022-12-07T04:52:24Z

Installables that point to a builtins.storePath or builtins.fetchClosure cannot be used from the command line due to not having a drvPath, yet they can be used if wrapped in a builtins:buildenv or any other derivation wrapper.

Describe the solution you'd like
Allow usage of installables (even flake references) without requiring them to have drvPath's.

Describe alternatives you've considered

Wrapping each such installable in a builtins:buildenv, which works.
I tried following the source. It seems that goals need a name, and for attribute/flakerefs it needs a drvPath to "name" the goal, but otherwise it looks like an artificial limitation.
nix build $(nix eval --impure .#some-attr-with-builtins-storePath) works, which is odd because one would think this could be done automatically

Priorities

Add 👍 to issues you find important.

The text was updated successfully, but these errors were encountered:

Ericson2314 · 2022-12-07T16:18:12Z

So concretely today we have

$ nix build --impure --expr 'builtins.storePath "/nix/store/4ywap29s7i8irjz95iz8hakvyc2836bf-neovim-0.7.0/bin/nvim"'
error: expression does not evaluate to a derivation (or a set or list of those)

yet

$ nix build /nix/store/4ywap29s7i8irjz95iz8hakvyc2836bf-neovim-0.7.0/bin/nvim

works fine.

Strictly speaking, the thing that doesn't work is still a bona fide installable in that the command line parsed successfully, but that it didn't work because the evaluation-based ones are very tricky over what sort of values they allow I think is a bit silly.

Ericson2314 · 2022-12-07T16:28:15Z

I think strings which are paths / placeholders (for CA derivation) that match their context should also be allowed.

To take our example above:

nix-repl> :p builtins.getContext (builtins.storePath "/nix/store/4ywap29s7i8irjz95iz8hakvyc2836bf-neovim-0.7.0/bin/nvim")
{ "/nix/store/4ywap29s7i8irjz95iz8hakvyc2836bf-neovim-0.7.0" = { path = true; }; }

the context and string match, so this should be valid.

For another example:

nix-repl> hello.out.outPath
"/nix/store/jppfl2bp1zhx8sgs2mgifmsx6dv16mv2-hello-2.12"

nix-repl> :p builtins.getContext hello.out.outPath
{ "/nix/store/c7jrxqjhdda93lhbkanqfs07x2bzazbm-hello-2.12.drv" = { outputs = [ "out" ]; }; }

the string matches the specified single output of that derivation, so it should also be valid.

For a final CA example:

nix-repl> (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
"/1a08j26xqc0zm8agps8anxpjji410yvsx4pcgyn4bfan1ddkx2g0"

nix-repl> :p builtins.getContext (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
{ "/nix/store/qc645pyf9wl37c6qvqzaqkwsm1gp48al-hello-2.12.drv" = { outputs = [ "out" ]; }; }

the stting is not a path but a placeholder, however it also matches the context, and because it is a CA derivation we have no better option. This should also be valid.

roberth · 2022-12-09T12:57:02Z

This seems like a good step towards #6507 as well, which suggests relying on .outputs rather than .drvPath.

Ericson2314 · 2022-12-09T14:11:57Z

Yes, it definitely does also relate to that and RFC 92 forwards compat.

nixos-discourse · 2022-12-12T10:04:07Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-09-nix-team-meeting-minutes-15/23951/1

Ericson2314 · 2022-12-15T17:09:57Z

#7467 relates to this; we can better document this new feature once derived paths are documented.

toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath with some attributes needed by 'nix profile' etc. Preparation for NixOS#7417.

This makes 'nix build' work on paths (which will be copied to the store) and store paths (returned as is). E.g. the following flake output attributes can be built using 'nix build .#foo': foo = ./src; foo = self.outPath; foo = builtins.fetchTarball { ... }; foo = (builtins.fetchTree { .. }).outPath; foo = builtins.fetchTree { .. } + "/README.md"; foo = builtins.storePath /nix/store/...; Note that this is potentially risky, e.g. foo = /.; will cause Nix to try to copy the entire file system to the store. What doesn't work yet: foo = self; foo = builtins.fetchTree { .. }; because we don't handle attrsets with an outPath attribute in it yet, and foo = builtins.storePath /nix/store/.../README.md; since result symlinks have to point to a store path currently (rather than a file inside a store path). Fixes NixOS#7417.

edolstra · 2022-12-15T22:12:56Z

I've implemented this in edolstra@81a4516 on the lazy-trees branch. I'll probably create a separate PR for it.

Ericson2314 · 2022-12-15T22:31:29Z

@edolstra Excellent! Really glad to hear this! I will can then make a PR supporting a few cases (drv & output in string context with known output store path, drv & output in strong context with placeholder).

Ericson2314 · 2022-12-15T22:34:28Z

I see DerivedPathWithInfo; we were thinking alike here :).

toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath with some attributes needed by 'nix profile' etc. Preparation for NixOS#7417.

This makes 'nix build' work on paths (which will be copied to the store) and store paths (returned as is). E.g. the following flake output attributes can be built using 'nix build .#foo': foo = ./src; foo = self.outPath; foo = builtins.fetchTarball { ... }; foo = (builtins.fetchTree { .. }).outPath; foo = builtins.fetchTree { .. } + "/README.md"; foo = builtins.storePath /nix/store/...; Note that this is potentially risky, e.g. foo = /.; will cause Nix to try to copy the entire file system to the store. What doesn't work yet: foo = self; foo = builtins.fetchTree { .. }; because we don't handle attrsets with an outPath attribute in it yet, and foo = builtins.storePath /nix/store/.../README.md; since result symlinks have to point to a store path currently (rather than a file inside a store path). Fixes NixOS#7417.

tomberek · 2023-01-04T01:55:48Z

I'm working on making this available for both

attrsets with "outPath" keys
attrsets which are derivations and incorporating the meta.outputsToInstall logic (the complication here is that these could be missing a drvPath and the DerivedPaths structs currently support either only a raw path or outputs that include drvPath)

WIP: master...flox:nix:fix-7417

Ericson2314 · 2023-01-10T20:00:28Z

We need to handle a few more things before this is done.

Output path string contexts
Non-flake attributes (the code is super duplicated right now)
placeholders for CA derivations

Progress towards NixOS#7417

As discussed in NixOS#7417, it would be good to make more string values work as installables. That is to say, if an installable refers to a value, and the value is a string, it used to not work at all, since NixOS#7484, it works somewhat, and this PR make it work some more. The new cases that are added for `BuiltPath` contexts: - Fixed input- or content-addressed derivation: ``` nix-repl> hello.out.outPath "/nix/store/jppfl2bp1zhx8sgs2mgifmsx6dv16mv2-hello-2.12" nix-repl> :p builtins.getContext hello.out.outPath { "/nix/store/c7jrxqjhdda93lhbkanqfs07x2bzazbm-hello-2.12.drv" = { outputs = [ "out" ]; }; } The string matches the specified single output of that derivation, so it should also be valid. - Floating content-addressed derivation: ``` nix-repl> (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath "/1a08j26xqc0zm8agps8anxpjji410yvsx4pcgyn4bfan1ddkx2g0" nix-repl> :p builtins.getContext (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath { "/nix/store/qc645pyf9wl37c6qvqzaqkwsm1gp48al-hello-2.12.drv" = { outputs = [ "out" ]; }; } ``` The string is not a path but a placeholder, however it also matches the context, and because it is a CA derivation we have no better option. This should also be valid. We may also want to think about richer attrset based values (also discussed in that issue and NixOS#6507), but this change "completes" our string-based building blocks, from which the others can be desugared into or at least described/document/taught in terms of. Progress towards NixOS#7417

Ericson2314 · 2023-02-27T14:07:23Z

#7910 covers some issue of drvPath alone as an installable.

As discussed in NixOS#7417, it would be good to make more string values work as installables. That is to say, if an installable refers to a value, and the value is a string, it used to not work at all, since NixOS#7484, it works somewhat, and this PR make it work some more. The new cases that are added for `BuiltPath` contexts: - Fixed input- or content-addressed derivation: ``` nix-repl> hello.out.outPath "/nix/store/jppfl2bp1zhx8sgs2mgifmsx6dv16mv2-hello-2.12" nix-repl> :p builtins.getContext hello.out.outPath { "/nix/store/c7jrxqjhdda93lhbkanqfs07x2bzazbm-hello-2.12.drv" = { outputs = [ "out" ]; }; } The string matches the specified single output of that derivation, so it should also be valid. - Floating content-addressed derivation: ``` nix-repl> (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath "/1a08j26xqc0zm8agps8anxpjji410yvsx4pcgyn4bfan1ddkx2g0" nix-repl> :p builtins.getContext (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath { "/nix/store/qc645pyf9wl37c6qvqzaqkwsm1gp48al-hello-2.12.drv" = { outputs = [ "out" ]; }; } ``` The string is not a path but a placeholder, however it also matches the context, and because it is a CA derivation we have no better option. This should also be valid. We may also want to think about richer attrset based values (also discussed in that issue and NixOS#6507), but this change "completes" our string-based building blocks, from which the others can be desugared into or at least described/document/taught in terms of. Progress towards NixOS#7417

As discussed in NixOS#7417, it would be good to make more string values work as installables. That is to say, if an installable refers to a value, and the value is a string, it used to not work at all, since NixOS#7484, it works somewhat, and this PR make it work some more. The new cases that are added for `BuiltPath` contexts: - Fixed input- or content-addressed derivation: ``` nix-repl> hello.out.outPath "/nix/store/jppfl2bp1zhx8sgs2mgifmsx6dv16mv2-hello-2.12" nix-repl> :p builtins.getContext hello.out.outPath { "/nix/store/c7jrxqjhdda93lhbkanqfs07x2bzazbm-hello-2.12.drv" = { outputs = [ "out" ]; }; } The string matches the specified single output of that derivation, so it should also be valid. - Floating content-addressed derivation: ``` nix-repl> (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath "/1a08j26xqc0zm8agps8anxpjji410yvsx4pcgyn4bfan1ddkx2g0" nix-repl> :p builtins.getContext (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath { "/nix/store/qc645pyf9wl37c6qvqzaqkwsm1gp48al-hello-2.12.drv" = { outputs = [ "out" ]; }; } ``` The string is not a path but a placeholder, however it also matches the context, and because it is a CA derivation we have no better option. This should also be valid. We may also want to think about richer attrset based values (also discussed in that issue and NixOS#6507), but this change "completes" our string-based building blocks, from which the others can be desugared into or at least described/document/taught in terms of. Progress towards NixOS#7417 Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

tomberek added the feature Feature request or proposal label Dec 7, 2022

Ericson2314 added this to Nix team Dec 7, 2022

thufschmitt moved this to ⏰ Postponed in Nix team Dec 9, 2022

tomberek self-assigned this Dec 9, 2022

Ericson2314 mentioned this issue Dec 15, 2022

--derivation considered harmful, remove it in lieu of ^ and new stuff #7261

Open

5 tasks

edolstra added a commit to edolstra/nix that referenced this issue Dec 15, 2022

Merge toDerivations() into toDerivedPaths()

210cd8c

toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath with some attributes needed by 'nix profile' etc. Preparation for NixOS#7417.

edolstra added a commit to edolstra/nix that referenced this issue Dec 20, 2022

Merge toDerivations() into toDerivedPaths()

845fc3f

toDerivedPaths() now returns DerivedPathWithInfo, which is DerivedPath with some attributes needed by 'nix profile' etc. Preparation for NixOS#7417.

edolstra mentioned this issue Dec 20, 2022

InstallableFlake::toDerivedPaths(): Support paths and store paths #7484

Merged

edolstra closed this as completed in #7484 Jan 10, 2023

github-project-automation bot moved this from ⏰ Postponed to ✅ Assigned in Nix team Jan 10, 2023

Ericson2314 reopened this Jan 10, 2023

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Jan 14, 2023

Make more string values work as installables

738c727

Progress towards NixOS#7417

Ericson2314 mentioned this issue Jan 14, 2023

Make more string values work as installables #7601

Merged

7 tasks

Ericson2314 added a commit to obsidiansystems/nix that referenced this issue Jan 16, 2023

Make more string values work as installables

7e946b5

Progress towards NixOS#7417

Ericson2314 mentioned this issue Jan 16, 2023

Get rid of .drv special-casing for store path installable #7600

Merged

7 tasks

thufschmitt added the idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. label Feb 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow installables without drvPath #7417

allow installables without drvPath #7417

tomberek commented Dec 7, 2022 •

edited

Loading

Ericson2314 commented Dec 7, 2022

Ericson2314 commented Dec 7, 2022 •

edited

Loading

roberth commented Dec 9, 2022

Ericson2314 commented Dec 9, 2022

nixos-discourse commented Dec 12, 2022

Ericson2314 commented Dec 15, 2022

edolstra commented Dec 15, 2022

Ericson2314 commented Dec 15, 2022 •

edited

Loading

Ericson2314 commented Dec 15, 2022

tomberek commented Jan 4, 2023 •

edited

Loading

Ericson2314 commented Jan 10, 2023

Ericson2314 commented Feb 27, 2023

allow installables without drvPath #7417

allow installables without drvPath #7417

Comments

tomberek commented Dec 7, 2022 • edited Loading

Ericson2314 commented Dec 7, 2022

Ericson2314 commented Dec 7, 2022 • edited Loading

roberth commented Dec 9, 2022

Ericson2314 commented Dec 9, 2022

nixos-discourse commented Dec 12, 2022

Ericson2314 commented Dec 15, 2022

edolstra commented Dec 15, 2022

Ericson2314 commented Dec 15, 2022 • edited Loading

Ericson2314 commented Dec 15, 2022

tomberek commented Jan 4, 2023 • edited Loading

Ericson2314 commented Jan 10, 2023

Ericson2314 commented Feb 27, 2023

tomberek commented Dec 7, 2022 •

edited

Loading

Ericson2314 commented Dec 7, 2022 •

edited

Loading

Ericson2314 commented Dec 15, 2022 •

edited

Loading

tomberek commented Jan 4, 2023 •

edited

Loading