Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get electrs source tarball with gpg verified sha256 and corresponding helper script #156

Merged
merged 2 commits into from
Apr 26, 2020

Conversation

nixbitcoin
Copy link
Member

This PR changes the electrs expression to fetch sources from the releases archive with fetchurl instead of fetchFromGitHub. This allows us to calculate the sha256 ahead of time and compare it with Roman Zeyde's gpg-signed git tags. I have provided a helper script to automate the process.

Overall this should improve expression integrity, and I plan to implement this for all our packages where gpg verification is possible.

@erikarvstedt
Copy link
Collaborator

Here are a few patches that I think are sensible.

What was the reason for the cargoDepsHook? Removing it produces the same build result with the same Cargo output.

@nixbitcoin nixbitcoin force-pushed the electrs-sha256 branch 2 times, most recently from f1e367a to b47b785 Compare April 16, 2020 09:00
@nixbitcoin
Copy link
Member Author

Wow, thank you. I learned so much from your commits. Truly much better with your changes.

The reason for cargoDepsHook was that I couldn't nixops deploy without it. Nixops was looking for Cargo.lock in source instead of in electrs-v0.8.3. So I symlinked them. Check if you can reproduce this error. If no, I will include a0fb625 which I have left out of my merge for now.

@nixbitcoin
Copy link
Member Author

Re: 90f3f35
Would it be possible for keybase.io or an MITM to include 2 PGP keys, one fake and one real, pass our gpg --list-keys check, and then verify the tag with fake key?

@nixbitcoin
Copy link
Member Author

I've added two commits 70363ae and 2132bcc which I feel provide cosmetic improvements. Please review.

@erikarvstedt
Copy link
Collaborator

erikarvstedt commented Apr 16, 2020

Re: 90f3f35
Yes, that's indeed possible. Great catch.
Here's a patch:

# Verify fingerprint and check that only one key was imported
[[ $(gpg --list-keys --with-colons $fingerprint) == $(gpg --list-keys --with-colons) ]]

Edit: I guess it's best to simply do it like this.

@erikarvstedt
Copy link
Collaborator

erikarvstedt commented Apr 16, 2020

Regarding cargoDepsHook: Nixops has no effect on the electrs derivation. But still, examples/deploy-nixops.sh runs fine without the cargoDepsHook.

@nixbitcoin
Copy link
Member Author

Regarding cargoDepsHook: Nixops has no effect on the electrs derivation. But still, examples/deploy-nixops.sh runs fine without the cargoDepsHook.

I get this when removing cargoDepsHook and running nixops deploy

source root is electrs-0.8.3
unpacking source archive /nix/store/zjjc6qbkxy62kx6m2a1dwz11cx7csvq0-electrs-0.8.3-vendor.tar.gz
diff: source/Cargo.lock: No such file or directory

ERROR: cargoSha256 is out of date

Cargo.lock is not the same in electrs-0.8.3-vendor.tar.gz

To fix the issue:
1. Use "1111111111111111111111111111111111111111111111111111" as the cargoSha256 value
2. Build the derivation and wait it to fail with a hash mismatch
3. Copy the 'got: sha256:' value back into the cargoSha256 field

nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix { inherit pkgs; }' runs fine with

...
Validating consistency between /tmp/nix-build-electrs-0.8.3.drv-0/electrs-0.8.3/Cargo.lock and /tmp/nix-build-electrs-0.8.3.drv-0/electrs-0.8.3-vendor.tar.gz/Cargo.lock
...

@nixbitcoin
Copy link
Member Author

Re: 90f3f35
Yes, that's indeed possible. Great catch.
Here's a patch:

# Verify fingerprint and check that only one key was imported
[[ $(gpg --list-keys --with-colons $fingerprint) == $(gpg --list-keys --with-colons) ]]

Edit: I guess it's best to simply do it like this.

Implemented in 141cad1

@erikarvstedt
Copy link
Collaborator

I can't reproduce the error when removing cargoDepsHook.
Can you share a minimal repro, so I can look into this?

@nixbitcoin
Copy link
Member Author

I'm deploying from the /examples folder on master branch. I have my configuration.nix, hardware-configuration.nix and nixops directory in /examples.

When using your version without cargoDepsHook:

cd ~/nix-bitcoin/examples
nix-shell
nixops deploy

I get error

building '/nix/store/5cx2j8bdcly1im82dwb7xbcsc7c6hw1f-electrs-0.8.3.drv'...
unpacking sources
unpacking source archive /nix/store/8wza0s4plclrphddlzjvm2a3zzs1r8xp-v0.8.3.tar.gz
source root is electrs-0.8.3
unpacking source archive /nix/store/zjjc6qbkxy62kx6m2a1dwz11cx7csvq0-electrs-0.8.3-vendor.tar.gz
diff: source/Cargo.lock: No such file or directory

ERROR: cargoSha256 is out of date

Cargo.lock is not the same in electrs-0.8.3-vendor.tar.gz

To fix the issue:
1. Use "1111111111111111111111111111111111111111111111111111" as the cargoSha256 value
2. Build the derivation and wait it to fail with a hash mismatch
3. Copy the 'got: sha256:' value back into the cargoSha256 field

builder for '/nix/store/5cx2j8bdcly1im82dwb7xbcsc7c6hw1f-electrs-0.8.3.drv' failed with exit code 1

nix show-derivation /nix/store/5cx2j8bdcly1im82dwb7xbcsc7c6hw1f-electrs-0.8.3.drv returns

{
  "/nix/store/5cx2j8bdcly1im82dwb7xbcsc7c6hw1f-electrs-0.8.3.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/sf7c3lcyjy2g4nyggv2nd58psh5rw5k2-electrs-0.8.3"
      }
    },
    "inputSrcs": [
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh",
      "/nix/store/nk6b2ckznjic5wj8ddw0wgdrn4mbz3lg-patch-registry-deps",
      "/nix/store/q0hmsd1h8bph16h58w80nafjg48yqdbw-fetchcargo-default-config.toml"
    ],
    "inputDrvs": {
      "/nix/store/7cipdvw2s5sz1px8x3hv41j40753969s-bash-4.4-p23.drv": [
        "out"
      ],
      "/nix/store/91ykhcvv5ngmn4clprglgv09c2ljnqcy-clang-wrapper-7.1.0.drv": [
        "out"
      ],
      "/nix/store/9m2g8agw9vg7rksjh6xjp0bwm68i04m2-git-2.25.1.drv": [
        "out"
      ],
      "/nix/store/a0bln04r79irdgx9nn673fh25yhf57x4-rustc-1.41.0.drv": [
        "out"
      ],
      "/nix/store/ay68yv59w82yy0438b5f9hh0r4bj8ps4-electrs-0.8.3-vendor.tar.gz.drv": [
        "out"
      ],
      "/nix/store/brpgrg9c3b2w6345b4p4mh09p38v6q0p-nss-cacert-3.49.2.drv": [
        "out"
      ],
      "/nix/store/d99rxrfh283phiv4mjsvgav3n8mx6wmj-clang-7.1.0.drv": [
        "lib"
      ],
      "/nix/store/hzd53xaaqkh0jib90hv31ixjgxrlppi9-stdenv-linux.drv": [
        "out"
      ],
      "/nix/store/m7q5wacivpbpb574iww0alflmhmfxv7c-v0.8.3.tar.gz.drv": [
        "out"
      ],
      "/nix/store/x1bf5vxjf3wbrv1m3s6qcrdkj2z6q4i8-cargo-1.41.0.drv": [
        "out"
      ],
      "/nix/store/zv92ycs354mzyjdczsypvx0c69jaammx-gcc-wrapper-9.2.0.drv": [
        "out"
      ]
    },
    "platform": "x86_64-linux",
    "builder": "/nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "env": {
      "LIBCLANG_PATH": "/nix/store/q53f79a4w3hw0z9vl43mb18ai75bx9b5-clang-7.1.0-lib/lib",
      "PKG_CONFIG_ALLOW_CROSS": "0",
      "buildInputs": "/nix/store/l7rrml4ph8agz3z7dazbn2d6iq45s9v0-clang-wrapper-7.1.0",
      "buildPhase": "runHook preBuild\n\n(\nset -x\nenv \\\n  \"CC_x86_64-unknown-linux-gnu\"=\"/nix/store/1kn7fi3hhi33jms3113riyzwyn2yqpqd-gcc-wrapper-9.2.0/bin/cc\" \\\n  \"CXX_x86_64-unknown-linux-gnu\"=\"/nix/store/1kn7fi3hhi33jms3113riyzwyn2yqpqd-gcc-wrapper-9.2.0/bin/c++\" \\\n  \"CC_x86_64-unknown-linux-gnu\"=\"/nix/store/1kn7fi3hhi33jms3113riyzwyn2yqpqd-gcc-wrapper-9.2.0/bin/cc\" \\\n  \"CXX_x86_64-unknown-linux-gnu\"=\"/nix/store/1kn7fi3hhi33jms3113riyzwyn2yqpqd-gcc-wrapper-9.2.0/bin/c++\" \\\n  cargo build \\\n    --release \\\n    --target x86_64-unknown-linux-gnu \\\n    --frozen \n)\n\n# rename the output dir to a architecture independent one\nmapfile -t targets < <(find \"$NIX_BUILD_TOP\" -type d | grep 'target/x86_64-unknown-linux-gnu/release$')\nfor target in \"${targets[@]}\"; do\n  rm -rf \"$target/../../release\"\n  ln -srf \"$target\" \"$target/../../\"\ndone\n\nrunHook postBuild\n",
      "builder": "/nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23/bin/bash",
      "cargoDeps": "/nix/store/zjjc6qbkxy62kx6m2a1dwz11cx7csvq0-electrs-0.8.3-vendor.tar.gz",
      "cargoSha256": "1x88zj7p4i7pfb25ch1a54sawgimq16bfcsz1nmzycc8nbwbf493",
      "checkPhase": "runHook preCheck\necho \"Running cargo cargo test -- ${checkFlags} ${checkFlagsArray+${checkFlagsArray[@]}}\"\ncargo test -- ${checkFlags} ${checkFlagsArray+\"${checkFlagsArray[@]}\"}\nrunHook postCheck\n",
      "configureFlags": "",
      "configurePhase": "runHook preConfigure\nrunHook postConfigure\n",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "doCheck": "1",
      "doInstallCheck": "",
      "installPhase": "runHook preInstall\nmkdir -p $out/bin $out/lib\n\nfind $releaseDir \\\n  -maxdepth 1 \\\n  -type f \\\n  -executable ! \\( -regex \".*\\.\\(so.[0-9.]+\\|so\\|a\\|dylib\\)\" \\) \\\n  -print0 | xargs -r -0 cp -t $out/bin\nfind $releaseDir \\\n  -maxdepth 1 \\\n  -regex \".*\\.\\(so.[0-9.]+\\|so\\|a\\|dylib\\)\" \\\n  -print0 | xargs -r -0 cp -t $out/lib\nrmdir --ignore-fail-on-non-empty $out/lib $out/bin\nrunHook postInstall\n",
      "name": "electrs-0.8.3",
      "nativeBuildInputs": "/nix/store/7gbcvdzx01bnzz9n3r5v1fij7cflapda-nss-cacert-3.49.2 /nix/store/3gjaqccy2y5slhjp4lhyxf8fqs594097-git-2.25.1 /nix/store/lzklv731zd9pvklbi9bv30gjgd8yf1ww-cargo-1.41.0 /nix/store/ayrmp5x5rgqqd0gs98fxha9asdi91nzp-rustc-1.41.0",
      "out": "/nix/store/sf7c3lcyjy2g4nyggv2nd58psh5rw5k2-electrs-0.8.3",
      "outputs": "out",
      "patchRegistryDeps": "/nix/store/nk6b2ckznjic5wj8ddw0wgdrn4mbz3lg-patch-registry-deps",
      "patches": "",
      "pname": "electrs",
      "postUnpack": "eval \"$cargoDepsHook\"\n\nunpackFile \"$cargoDeps\"\ncargoDepsCopy=$(stripHash $cargoDeps)\n\n\nmkdir .cargo\nconfig=\"$(pwd)/$cargoDepsCopy/.cargo/config\";\nif [[ ! -e $config ]]; then\n  config=/nix/store/q0hmsd1h8bph16h58w80nafjg48yqdbw-fetchcargo-default-config.toml;\nfi;\nsubstitute $config .cargo/config \\\n  --subst-var-by vendor \"$(pwd)/$cargoDepsCopy\"\n\ncat >> .cargo/config <<'EOF'\n[target.\"x86_64-unknown-linux-gnu\"]\n\"linker\" = \"/nix/store/1kn7fi3hhi33jms3113riyzwyn2yqpqd-gcc-wrapper-9.2.0/bin/cc\"\n\nEOF\n\nexport RUST_LOG=\nif ! diff source/Cargo.lock $cargoDepsCopy/Cargo.lock ; then\n  echo\n  echo \"ERROR: cargoSha256 is out of date\"\n  echo\n  echo \"Cargo.lock is not the same in $cargoDepsCopy\"\n  echo\n  echo \"To fix the issue:\"\n  echo '1. Use \"1111111111111111111111111111111111111111111111111111\" as the cargoSha256 value'\n  echo \"2. Build the derivation and wait it to fail with a hash mismatch\"\n  echo \"3. Copy the 'got: sha256:' value back into the cargoSha256 field\"\n  echo\n\n  exit 1\nfi\nunset cargoDepsCopy\n",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "releaseDir": "target/x86_64-unknown-linux-gnu/release",
      "src": "/nix/store/8wza0s4plclrphddlzjvm2a3zzs1r8xp-v0.8.3.tar.gz",
      "stdenv": "/nix/store/0w5454az7vwcq60yqvcsv8fs9q7r4zrx-stdenv-linux",
      "strictDeps": "",
      "system": "x86_64-linux",
      "version": "0.8.3"
    }
  }
}

@erikarvstedt
Copy link
Collaborator

erikarvstedt commented Apr 17, 2020

Ah, the error appears when building the electrs pkg pinned to nixpkgs-unstable that's used in the modules:

nix-build --no-out-link -A pinned.electrs

This bug has been fixed in the current unstable channel, so I'd propose to just update the channel.

nixbitcoin added a commit to nixbitcoin/nix-bitcoin that referenced this pull request Apr 17, 2020
Specifically to advance
fort-nix#156 without
cargoDepsHook
@erikarvstedt
Copy link
Collaborator

LGTM. Can you squash the fixup commits and put the Update nixpkgs commit first?

@nixbitcoin
Copy link
Member Author

With Update nixpkgs I keep on getting

   Compiling librocksdb-sys v5.18.3
  CXX      libbitcoin_server_a-block_proof.o
  CXX      libbitcoin_server_a-blockfilter.o
  CXX      libbitcoin_server_a-init.o
  CXX      libbitcoin_server_a-dbwrapper.o
  CXX      libbitcoin_server_a-mainchainrpc.o
  CXX      libbitcoin_server_a-merkleblock.o
  CXX      libbitcoin_server_a-miner.o
  CXX      libbitcoin_server_a-net.o
  CXX      libbitcoin_server_a-net_processing.o
  CXX      libbitcoin_server_a-noui.o
  CXX      libbitcoin_server_a-outputtype.o
error: failed to run custom build command for `librocksdb-sys v5.18.3`

Caused by:
  process didn't exit successfully: `/tmp/nix-build-electrs-0.8.3.drv-0/electrs-0.8.3/target/release/build/librocksdb-sys-4b3cd55de6da731d/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=rocksdb/
cargo:rerun-if-changed=snappy/
cargo:rerun-if-changed=lz4/
cargo:rerun-if-changed=zstd/
cargo:rerun-if-changed=zlib/
cargo:rerun-if-changed=bzip2/

--- stderr
rocksdb/include/rocksdb/c.h:65:10: fatal error: 'stdarg.h' file not found
rocksdb/include/rocksdb/c.h:65:10: fatal error: 'stdarg.h' file not found, err: true
thread 'main' panicked at 'unable to generate rocksdb bindings: ()', /tmp/nix-build-electrs-0.8.3.drv-0/electrs-0.8.3-vendor.tar.gz/librocksdb-sys/build.rs:34:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I tried changing clang to a newer version with

-{ lib, rustPlatform, clang, llvmPackages, fetchurl, pkgs }:
+{ lib, rustPlatform, llvmPackages_10, fetchurl, pkgs }:
 rustPlatform.buildRustPackage rec {
   pname = "electrs";
   version = "0.8.3";
@@ -10,8 +10,8 @@ rustPlatform.buildRustPackage rec {
   };
 
   # Needed for librocksdb-sys
-  buildInputs = [ clang ];
-  LIBCLANG_PATH = "${llvmPackages.libclang}/lib";
+  buildInputs = [ llvmPackages_10.clang ];
+  LIBCLANG_PATH = "${llvmPackages_10.libclang}/lib";

But still got the error

I think it's somehow related to romanz/electrs#226 and NixOS/nixpkgs@dc3c338#diff-0c4100e0613807f5600eb55c5446b20e but can't figure out how. Maybe you have an idea.

@nixbitcoin nixbitcoin force-pushed the electrs-sha256 branch 2 times, most recently from 82e5d87 to 7f8e452 Compare April 19, 2020 13:06
… helper script

move script to pkg dir, add hint to script in pkg def

remove unneeded script deps

add extended bash error checking

rename DIR -> TMPDIR

remove TMPDIR on exit

strip whitespace, simplify comments

gpg2 -> gpg

latesttagelectrs -> latest

tmpdir: don't use XDG_RUNTIME_DIR

XDG_RUNTIME_DIR is often in RAM and shouldn't be used for larger
workloads like repo downlaods

verify fingerprint of the imported key

remove trailing '-' in output

simplify output

Hide --fetch-key output

Output is not relevant to user, looks better without it

More accurately describe ./get-sha256 function

User might think that ./get-sha256 automatically updates sha256 in default.nix

Fetch key from sks keyservers instead of keybase.io

Using --recv-key simplifies getting the right key, and only the
right key, greatly. I try to refrain from using sks keyservers,
but the certificate spamming attack shouldn't be an issue in this
case because we create a temporary keychain just for the
verificaiton.

remove unneeded cargoDepsHook

Make clang nativeBuildInput instead of buildInput
@nixbitcoin
Copy link
Member Author

nixbitcoin commented Apr 26, 2020

I was finally able to fix the issue by making clang a nativeBuildInput instead of a buildInput.

LGTM. Can you squash the fixup commits and put the Update nixpkgs commit first?

Done

@erikarvstedt
Copy link
Collaborator

erikarvstedt commented Apr 26, 2020

The test currently times out because of the channel update.
Can you copy the branch to this repo (fort-nix/nix-bitcoin) to enable cachix uploading in Travis? Restarting the test should then fix the timeout.

Copy link
Collaborator

@erikarvstedt erikarvstedt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test runs fine locally.

Copy link
Member

@jonasnick jonasnick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 1acb22a

I pushed the branch to this repo to enable cachix pushes and fix travis.

I was finally able to fix the issue by making clang a nativeBuildInput instead of a buildInput.

Ouch, this probably took a while.

@jonasnick jonasnick merged commit 199b9bf into fort-nix:master Apr 26, 2020
@nixbitcoin nixbitcoin deleted the electrs-sha256 branch March 3, 2021 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants