naersk rewrites Cargo.toml to something unparseable

[GoldsteinE]

I tried to install dylint with these packages:

(naersk'.buildPackage {
  root = dylint;  # where `dylint` is a flake input pointing to `github:trailofbits/dylint/v2.0.12`
  cargoBuildOptions = x: x ++ [ "-p" "cargo-dylint" ];
  cargoTestOptions = x: x ++ [ "-p" "cargo-dylint" ];
})
(naersk'.buildPackage {
  root = dylint;
  cargoBuildOptions = x: x ++ [ "-p" "dylint-link" ];
  cargoTestOptions = x: x ++ [ "-p" "dylint-link" ];
})

I get this error:

error: failed to parse manifest at `/build/dummy-src/Cargo.toml`

Caused by:
  could not parse input as TOML

Caused by:
  TOML parse error at line 3, column 39
    |
  3 | metadata = { dylint = { libraries = [ {"path":"examples/*/*"} ] } }
    |                                       ^
  Unexpected `{`
  Expected `]`

Indeed, TOML in dummy-src looks like this:

[workspace]
members = [ "cargo-dylint", "dylint", "dylint-link", "internal", "utils/linting", "utils/testing" ]
metadata = { dylint = { libraries = [ {"path":"examples/*/*"} ] } }

While the actual TOML from the repo is correct:

# smoelius: `dylint_driver` is in its own workspace because it requires specific Rust components.

[workspace]
members = [
    "cargo-dylint",
    # "driver",
    "dylint",
    "dylint-link",
    "internal",
    "utils/linting",
    "utils/testing",
]

[workspace.metadata.dylint]
libraries = [
    { path = "examples/*/*" },
]

[GoldsteinE]

https://github.com/nix-community/naersk/blob/master/builtins/to-toml.nix#L51-L53

It seems like this is the bug: it’s incorrect to just toJSON every val in list of attrs. The correct output should probably be

[[workspace.metadata.dylint.libraries]]
path = "examples/*/*"

[Patryk27]

I see - thanks for investigating, I'll try fixing it 🙂

[GoldsteinE]

As implementing full TOML serializer in Nix is complicated and further bugs may arise, maybe there is value in providing usePureToTOML = false; option to use external TOML serializer?

[Xe]

I ran into this too:

[package.metadata.generate-rpm]
assets = [
  { source = "target/release/xn--ts9h", dest = "/usr/bin/🥺", mode = "6755"},
  { source = "README.md", dest = "/usr/share/doc/🥺/README", mode = "644", doc = true},
  { source = "LICENSE", dest = "/usr/share/doc/🥺/LICENSE", mode = "644", doc = true},
  { source = "🥺.8", dest = "/usr/share/man/man8/🥺.8", mode = "644", doc = true},
]

Amazingly the emoji is NOT the issue!

[Pegasust]

Chiming in to the issue train, this is preventing us from compiling Leptos (in a workspace):

Original

[[workspace.metadata.leptos]]
## Workspace config
# The name used by wasm-bindgen/cargo-leptos for the JS/WASM bundle
name = "proto"
# Executable for the webserver
bin-package = "proto-web"
lib-package = "proto-front"
# ...

Processed:

 metadata = { leptos = [ {"assets-dir":"public","bin-default-features":false,"bin-features":["ssr"],"bin-package":"huawei-pro
to-web","browserquery":"defaults","end2end-cmd":"npx playwright test","env":"DEV","lib-default-features":false,"lib-features":["hydrate"],"
lib-package":"huawei-proto-front","name":"huawei-proto","reload-port":3001,"site-addr":"127.0.0.1:3000","site-pkg-dir":"pkg","site-root":"t
arget/huawei-proto","tailwind-config-file":"./frontend/huawei-proto-front/tailwind.config.js","tailwind-input-file":"./frontend/huawei-prot
o-front/tailwind.css"} ] }

I'm relatively new to the codebase. Why are we having a rewriting of Cargo.toml?

[Patryk27]

Why are we having a rewriting of Cargo.toml?

Naersk (and Crane as well, IIRC) have a two-step build process: first they build all of the dependencies (and cache the output), and later they build the actual application (utilizing prebuilt dependencies from the previous step); this allows for faster incremental rebuilds, since more frequently it's the user's application code that changes, not its dependencies.

This two-step approach requires generating a stripped version of the original Cargo.toml - i.e. if one is trying to build:

[[bin]]
path = "main.rs"

[dependencies]
/* something something */

... what Naersk builds first is just:

[dependencies]
/* something something */

... i.e. without any [[bin]], [[example]] etc.

I'll try to take a look some time this week, but for a hotfix, you can tell Naersk not to generate prebuilt dependencies:

naersk.buildPackage {
  # ...
  singleStep = true;
}

[crabdancing]

got this same issue when trying to package probe-rs:

{
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    naersk.url = "github:nix-community/naersk";
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    probe-rs = {
      url = "github:probe-rs/probe-rs";
      flake = false;
    };
  };

  outputs = { self, flake-utils, naersk, nixpkgs, probe-rs }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = (import nixpkgs) {
          inherit system;
        };

        naersk' = pkgs.callPackage naersk {};

      in rec {
        # For `nix build` & `nix run`:
        defaultPackage = naersk'.buildPackage {
          src = probe-rs;
        };

        # For `nix develop`:
        devShell = pkgs.mkShell {
          nativeBuildInputs = with pkgs; [ rustc cargo ];
        };
      }
    );
}

warning: updating lock file '/home/ada/Projects/probe-rs/flake.lock':
• Updated input 'probe-rs':
    'https://github.com/probe-rs/probe-rs?narHash=sha256-hKzb2osv2pam4FeETd%2BUZWWMVLkMhzjYs%2BNQBpWqOko%3D'
  → 'git+https://github.com/probe-rs/probe-rs?ref=refs/heads/master&rev=8f50cc4a96328fdc791efa473794cb50bcee0d00' (2023-07-05)
error: builder for '/nix/store/j2sbf0vbnyzlahkqm34jnazbhxcvjwb7-rust-workspace-deps-unknown.drv' failed with exit code 101;
       last 10 log lines:
       > Caused by:
       >   could not parse input as TOML
       >
       > Caused by:
       >   TOML parse error at line 97, column 64
       >      |
       >   97 | metadata = { release = { "pre-release-replacements" = [ {"file":"../CHANGELOG.md","replace":"## [Unreleased]\n\n## [{{version}}]\n\nReleased {{date}}","search":"## \\[Unreleased\\]"}, {"file":"../CHANGELOG.md","replace":"[unreleased]: https://github.com/probe-rs/probe-rs/compare/v{{version}}...master\n[{{version}}]: https://github.com/probe-rs/probe-rs/compare/v$1...v{{version}}","search":"\\[unreleased\\]: https://github.com/probe-rs/probe-rs/compare/v([a-z0-9.-]+)\\.\\.\\.master"} ] } }
       >      |                                                                ^
       >   expected `.`, `=`
       > [naersk] cargo returned with exit code 101, exiting
       For full logs, run 'nix log /nix/store/j2sbf0vbnyzlahkqm34jnazbhxcvjwb7-rust-workspace-deps-unknown.drv'.
error: 1 dependencies of derivation '/nix/store/vwdgl4xpmj5k9cc2gbvsdhpc51g7baz0-rust-workspace-unknown.drv' failed to build