Completely drop stack.yaml and sha256map.nix #3328

Anviking · 2022-06-13T18:44:16Z

It seems stack.yaml and sha256map.nix can just be removed and replaced with checkCabalProject and --sha256 lines in cabal.project

Tested manually

Sanity checking that it works similarly to with the stack and sha256map-based approach:

Revision in cabal.project changed

$ ./nix/regenerate.sh
Changes required to cabal.project
--- cabal.project.old	2022-06-13 20:35:27.437617302 +0200
+++ cabal.project	2022-06-13 20:36:37.049474850 +0200
@@ -90,7 +90,7 @@
     type: git
     location: https://github.com/input-output-hk/cardano-addresses
     tag: 4ff9c4022d4d257053c96509f20c9d931ab8f57b
-    --sha256: 0iqzppqqb1xqnp55yd0xil9qfy8fz8474w834nmfg2w692bynyb9
+    --sha256: 0q93a2r4v8d3nfvix6yjmbh3v77hz2hpp6p11krklvnvw7h00lfk
     subdir: command-line
             core

(exit code 1)

Invalid SHA256 hash

$ ./nix/regenerate.sh
Changes required to cabal.project
--- cabal.project.old	2022-06-13 20:42:25.934613264 +0200
+++ cabal.project	2022-06-13 20:43:38.737611148 +0200
@@ -299,7 +299,7 @@
      type: git
      location: https://github.com/haskell-works/hw-aeson
      tag: d99d2f3e39a287607418ae605b132a3deb2b753f
-     --sha256: 0vxqcwjg9q37wbwi27y9ba5163lzfz51f1swbi0rp681yg63zvn4
+     --sha256: 1vxqcwjg9q37wbwi27y9ba5163lzfz51f1swbi0rp681yg63zvn4

(exit code 1)

Invalid SHA256 hash in `cabal.project` + `direnv reload`

The corresponding change on master would be having an incorrect hash in sha256map.nix and calling direnv reload. And if you on master changed a revision in stack.yaml, direnv reload would still work, only ./nix/regenerate.sh wouldn't. Because nix shell now is derived directly from cabal.project without intermediate state like sha256map.nix, direnv reload now fails directly.

direnv reload                                                                                                                  (4s 207ms)
direnv: loading ~/IOHK/cardano-wallet/.envrc
direnv: using flake
warning: Git tree '/Users/anviking/IOHK/cardano-wallet' is dirty
trace: To make project.plan-nix for cardano-wallet a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for cardano-wallet entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
direnv: ([/usr/local/bin/direnv export fish]) is taking a while to execute. Use CTRL-C to give up.
error: hash mismatch in fixed-output derivation '/nix/store/icv2hd3bghfg80fvppdnjp0b0h1wji7x-hw-aeson-d99d2f3.drv':
         specified: sha256-xO4/zPMBmZtBXFwHF8p3nw4TilrJHxH54mfg9CRnuG8=
            got:    sha256-xO4/zPMBmZtBXFwHF8p3nw4TilrJHxH54mfg9CRnuO8=
error: 1 dependencies of derivation '/nix/store/493wcz8r85zbq5yaimm74ldnwz62m6in-source-repository-package.drv' failed to build
error: 1 dependencies of derivation '/nix/store/2ja2788s8ivgsv2mcrb2b9pdl7b894dx-cardano-wallet-plan-to-nix-pkgs.drv' failed to build
(use '--show-trace' to show detailed location information)
warning: Git tree '/Users/anviking/IOHK/cardano-wallet' is dirty
direnv: renewed cache
direnv: export +XDG_DATA_DIRS

Comments

Targets #3280

Issue Number

ADP-1744

Anviking · 2022-06-13T18:44:45Z

.buildkite/pipeline.yml

@@ -32,7 +32,6 @@ steps:
    key: nix
    commands:
      - './nix/regenerate.sh'
-      - 'nix build .#checkStackProject -o check-stack-project.sh && ./check-stack-project.sh'


This line now exists inside ./nix/regenerate.sh

Anviking · 2022-06-13T18:45:25Z

stack.yaml

-    - stubs/plutus-ghc-stub
-    - prettyprinter-configurable
-    - word-array
->>>>>>> adcfc0603 (Update node to 1.35.0-rc2)


Hm, had left this in a broken state anyway 😅

Anviking · 2022-06-13T18:51:23Z

nix/haskell.nix

@@ -110,10 +110,6 @@ haskell-nix: haskell-nix.cabalProject' [
        filter = haskell-nix.haskellSourceFilter;
      };

-      # materialized = ./materialized/plan-nix;
-
-      sha256map = import ./sha256map.nix;


Seems like the sha256map was unused anyway 🤷‍♂️ (unless there is some clever fallback)

No, it seems the use is somewhat "magical". On master there is no more reference to sha256map, yet if we remove this statement there we get:

direnv reload (12s 11ms) direnv: loading ~/IOHK/cardano-wallet/.envrc direnv: using flake warning: Git tree '/Users/anviking/IOHK/cardano-wallet' is dirty direnv: ([/usr/local/bin/direnv export fish]) is taking a while to execute. Use CTRL-C to give up. error: Cannot find Git revision '7497a29cb998721a9068d5725d49461f2bba0e7a' in ref 'master' of repository 'https://github.com/input-output-hk/optparse-applicative'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit. (use '--show-trace' to show detailed location information) warning: Git tree '/Users/anviking/IOHK/cardano-wallet' is dirty direnv: renewed cache direnv: export +XDG_DATA_DIRS

whereas it is redundant in this PR (at least with the --sha256 info in cabal.project)

Unisay · 2022-06-14T07:37:24Z

@Anviking I am afraid I don't understand the purpose of the sha256map.nix file. Could you explain it plz?

Anviking · 2022-06-14T13:41:24Z

I am afraid I don't understand the purpose of the sha256map.nix file. Could you explain it plz?

@Unisay As I understand it:

Haskell.nix needs to know the content hash before fetching anything from GitHub or similar for security reasons.

On master the hashes for some of dependencies like cardano-addresses (I guess non-hackage/LTS deps) are stored in sha256map.nix, and are automatically derived from stack.yaml using ./nix/regenerate.sh.

In #3315 (or rather the included #3293) stack.yaml is made redundant, but sha256map.nix still isn't. This PR goes the final step and removing sha256map.nix in favour of storing the hashes in cabal.project (it was a pleasant surprise that this seemed to "just work").

michaelpj · 2022-06-15T09:12:13Z

This PR goes the final step and removing sha256map.nix in favour of storing the hashes in cabal.project (it was a pleasant surprise that this seemed to "just work").

FWIW here are some reasons not to do this:

Your cabal.project has some non-cabal "nix stuff" in it
If sha256map is in a separate file you can write some scripts to update it, which is more annoying if it's in the cabal.project

michaelpj · 2022-06-15T09:12:59Z

But you should be able to generate without needing stack.yaml... not sure if there's a script already lying around for that.

Anviking · 2022-06-15T16:46:41Z

If sha256map is in a separate file you can write some scripts to update it, which is more annoying if it's in the cabal.project

You mean like

 nix build .#checkCabalProject -o check-cabal-project.sh && ./check-cabal-project.sh

which does update the sha256 lines, or is there a concrete example of what else you might want to do?

Your cabal.project has some non-cabal "nix stuff" in it

TBH it seems kind of neat to me to have the rev and --sha256 diffs in the same file rather than split across two when updating something, even though I have never worked with it in practice.

michaelpj · 2022-06-16T08:38:23Z

Tastes differ I guess :)

sevanspowell · 2022-06-17T01:23:59Z

I would normally be inclined just to do what Michael and the Plutus team are doing (safety in numbers), but I don't think it matters too much here. We can always switch to the sha256map if we need to, and this solution is less confusing for the rest of the team, as they've seen this syntax before, but have yet to be exposed to the sha256map.

Nice work Johannes, thank you!

sevanspowell · 2022-06-17T02:53:01Z

bors r+

3328: Completely drop stack.yaml and sha256map.nix r=sevanspowell a=Anviking - It seems `stack.yaml` and `sha256map.nix` can just be removed and replaced with [`checkCabalProject`](https://github.com/input-output-hk/iohk-nix/search?q=checkCabalProject) and `--sha256` lines in `cabal.project` ### Tested manually Sanity checking that it works similarly to with the stack and sha256map-based approach: #### Revision in cabal.project changed ``` $ ./nix/regenerate.sh Changes required to cabal.project --- cabal.project.old 2022-06-13 20:35:27.437617302 +0200 +++ cabal.project 2022-06-13 20:36:37.049474850 +0200 `@@` -90,7 +90,7 `@@` type: git location: https://github.com/input-output-hk/cardano-addresses tag: 4ff9c4022d4d257053c96509f20c9d931ab8f57b - --sha256: 0iqzppqqb1xqnp55yd0xil9qfy8fz8474w834nmfg2w692bynyb9 + --sha256: 0q93a2r4v8d3nfvix6yjmbh3v77hz2hpp6p11krklvnvw7h00lfk subdir: command-line core ``` (exit code 1) #### Invalid SHA256 hash ``` $ ./nix/regenerate.sh Changes required to cabal.project --- cabal.project.old 2022-06-13 20:42:25.934613264 +0200 +++ cabal.project 2022-06-13 20:43:38.737611148 +0200 `@@` -299,7 +299,7 `@@` type: git location: https://github.com/haskell-works/hw-aeson tag: d99d2f3e39a287607418ae605b132a3deb2b753f - --sha256: 0vxqcwjg9q37wbwi27y9ba5163lzfz51f1swbi0rp681yg63zvn4 + --sha256: 1vxqcwjg9q37wbwi27y9ba5163lzfz51f1swbi0rp681yg63zvn4 ``` (exit code 1)  #### Invalid SHA256 hash in `cabal.project` + `direnv reload` The corresponding change on master would be having an incorrect hash in `sha256map.nix` and calling `direnv reload`. And if you on master changed a revision in `stack.yaml`, `direnv reload` would still work, only `./nix/regenerate.sh` wouldn't. Because `nix shell` now is derived directly from `cabal.project` without intermediate state like sha256map.nix, `direnv reload` now fails directly. ```bash direnv reload (4s 207ms) direnv: loading ~/IOHK/cardano-wallet/.envrc direnv: using flake warning: Git tree '/Users/anviking/IOHK/cardano-wallet' is dirty trace: To make project.plan-nix for cardano-wallet a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'. trace: To materialize project.plan-nix for cardano-wallet entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'. direnv: ([/usr/local/bin/direnv export fish]) is taking a while to execute. Use CTRL-C to give up. error: hash mismatch in fixed-output derivation '/nix/store/icv2hd3bghfg80fvppdnjp0b0h1wji7x-hw-aeson-d99d2f3.drv': specified: sha256-xO4/zPMBmZtBXFwHF8p3nw4TilrJHxH54mfg9CRnuG8= got: sha256-xO4/zPMBmZtBXFwHF8p3nw4TilrJHxH54mfg9CRnuO8= error: 1 dependencies of derivation '/nix/store/493wcz8r85zbq5yaimm74ldnwz62m6in-source-repository-package.drv' failed to build error: 1 dependencies of derivation '/nix/store/2ja2788s8ivgsv2mcrb2b9pdl7b894dx-cardano-wallet-plan-to-nix-pkgs.drv' failed to build (use '--show-trace' to show detailed location information) warning: Git tree '/Users/anviking/IOHK/cardano-wallet' is dirty direnv: renewed cache direnv: export +XDG_DATA_DIRS ``` ### Comments Targets #3280 ### Issue Number ADP-1744  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2022-06-17T02:53:37Z

Build failed:

buildkite/cardano-wallet

3280: Update to cardano-node 1.35-rc2 r=Anviking a=Anviking ### Overview - Updates dependencies to target `cardano-node 1.35-rc2` - Adds support to make the wallet _mostly_ work in Babbage - Rework of nix setup: we now derive nix files from `cabal.project` rather than the weird `stack.yaml`-based approach we had before. ### Detailed todo-list - [x] Get nix happy with the new dependencies - [x] Make `cabal build all` work with modulo warnings - [x] Update to more recent revisions - [x] Fix compilation in tests again - [x] Make `cabal build all` work with _no_ warnings - [x] Fix puzzling persistent TH name shadowing warnings -> fixed by sam? - [x] Ensure existing tests pass - [x] unit - [x] Look closer at balanceTx golden unit failure after first seeing if balanceTx integration tests pass on both alonzo and Babbage - [x] Integration (on alonzo like on master) - [x] Was blocked by readTVar, fixed with input-output-hk/io-sim#6 (review) - [x] Fix failures relating to Babbage era -> fixed with a hack 9e2568e - [x] Fix cardano-wallet-core Aeson failures - [x] More closely review code-changes - [x] Clean up git commit history - [x] Flatten merge commits - [x] Squash back-and-forth nix and revision changes - [x] Rebase off master - [x] Ensure revisions in stack.yaml and cabal.project are updated consistently - Alternatively remove `stack.yaml` and ensure ./nix/regenerate.sh still works and that we know how to update sha256map.nix — #3328 but without the sha256map.nix - [x] Ensure CI is completely green: [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-3280), ADP-1888 - There may be CI specific problems with nix, windows and more (there probably are) - [x] fix hydra wine TH error e.g. `$(TH.lift =<< TH.runIO (readFile "app/foo.txt"))` - seems relevant input-output-hk/haskell.nix#1508 - seems gone now - [x] `Ambiguous module name` https://hydra.iohk.io/build/15469467 - https://input-output-rnd.slack.com/archives/C48HA39RS/p1654388106509579 - Bumping Haskell.nix again should help? - Didn't seem like it worked... - https://input-output-rnd.slack.com/archives/CAP8NM7N0/p1654828014483969 - [x] Remove-iserv problem https://hydra.iohk.io/build/15604435 - Update from Hamish https://input-output-rnd.slack.com/archives/CAP8NM7N0/p1655213059547499?thread_ts=1654828014.483969&cid=CAP8NM7N0 - [x] Docs shell - we don't need explicit emanote input in nix config - [x] Profiled shell: `recursionzmschemeszm5zi2zi2zi2zm6NC6xZZx2tI3826DrVhqI9v_DataziFunctorziFoldableziTH_zdfMakeBaseFunctorNamezuzdcmakeBaseFunctor_closure` https://hydra.iohk.io/build/15694577/nixlog/2/tail - [x] musl cli goldens https://hydra.iohk.io/build/15694618/nixlog/82 - [x] `hie-bios: hie-bios: this operation requires -fexternal-interpreter - [x] `Ambiguous module name 'Options.Applicative':` in HLS - now we at least have an ugly `-XPackageImport` workaround. - [x] Update node revision in readme.md ### Things we need eventually, but not strictly for this PR > - Update to final 1.35 tag once it is released - can be done in separate PR > - Run existing integration tests on both Alonzo and Babbage eras in CI > - Depend on cardano-addresses with aeson2 support merged, tagged and released > - Depend on a cardano-sl-x509 where aeson 2 support is merged to master > - Consider appropriateness Tx deserialisation hack 9e2568e Moved to jira tasks in https://input-output.atlassian.net/browse/ADP-1538 ### Comments Includes #3258, #3293  ### Issue Number ADP-1744  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Pawel Jakubas <jakubas.pawel@gmail.com> Co-authored-by: Samuel Evans-Powell <mail@sevanspowell.net> Co-authored-by: Jonathan Knowles <jonathan.knowles@iohk.io>

Anviking commented Jun 13, 2022

View reviewed changes

Anviking requested a review from sevanspowell June 13, 2022 18:45

Anviking commented Jun 13, 2022

View reviewed changes

Anviking self-assigned this Jun 13, 2022

Anviking mentioned this pull request Jun 14, 2022

Update to cardano-node 1.35-rc2 #3280

Merged

29 tasks

Anviking force-pushed the anviking/ADP-1744/node-1.35 branch from 3c62f15 to 4f62922 Compare June 14, 2022 18:20

Anviking force-pushed the anviking/ADP-1744/completely-drop-stack.yaml branch from 2b34f06 to 603f7bb Compare June 14, 2022 20:16

jonathanknowles force-pushed the anviking/ADP-1744/node-1.35 branch from df70ec0 to 4cbf834 Compare June 16, 2022 01:20

Replace nix/sha256map and stack.yaml with checkCabalProject

8eea1d3

Anviking force-pushed the anviking/ADP-1744/completely-drop-stack.yaml branch from 9298e17 to 8eea1d3 Compare June 16, 2022 12:38

sevanspowell approved these changes Jun 17, 2022

View reviewed changes

sevanspowell merged commit 0129872 into anviking/ADP-1744/node-1.35 Jun 17, 2022

sevanspowell deleted the anviking/ADP-1744/completely-drop-stack.yaml branch June 17, 2022 03:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Completely drop stack.yaml and sha256map.nix #3328

Completely drop stack.yaml and sha256map.nix #3328

Anviking commented Jun 13, 2022 •

edited

Loading

Anviking Jun 13, 2022

Anviking Jun 13, 2022 •

edited

Loading

Anviking Jun 13, 2022

Anviking Jun 14, 2022 •

edited

Loading

Unisay commented Jun 14, 2022

Anviking commented Jun 14, 2022 •

edited

Loading

michaelpj commented Jun 15, 2022

michaelpj commented Jun 15, 2022

Anviking commented Jun 15, 2022

michaelpj commented Jun 16, 2022

sevanspowell commented Jun 17, 2022

sevanspowell commented Jun 17, 2022

iohk-bors bot commented Jun 17, 2022

Completely drop stack.yaml and sha256map.nix #3328

Completely drop stack.yaml and sha256map.nix #3328

Conversation

Anviking commented Jun 13, 2022 • edited Loading

Tested manually

Revision in cabal.project changed

Invalid SHA256 hash

Invalid SHA256 hash in cabal.project + direnv reload

Comments

Issue Number

Anviking Jun 13, 2022

Choose a reason for hiding this comment

Anviking Jun 13, 2022 • edited Loading

Choose a reason for hiding this comment

Anviking Jun 13, 2022

Choose a reason for hiding this comment

Anviking Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

Unisay commented Jun 14, 2022

Anviking commented Jun 14, 2022 • edited Loading

michaelpj commented Jun 15, 2022

michaelpj commented Jun 15, 2022

Anviking commented Jun 15, 2022

michaelpj commented Jun 16, 2022

sevanspowell commented Jun 17, 2022

sevanspowell commented Jun 17, 2022

iohk-bors bot commented Jun 17, 2022

Anviking commented Jun 13, 2022 •

edited

Loading

Invalid SHA256 hash in `cabal.project` + `direnv reload`

Anviking Jun 13, 2022 •

edited

Loading

Anviking Jun 14, 2022 •

edited

Loading

Anviking commented Jun 14, 2022 •

edited

Loading