Pantry does not work with git annex #4579

dbaynard · 2019-02-10T17:36:13Z

General summary/comments (optional)

Pantry does not support a git repo which uses git-annex.

Steps to reproduce

For example, using master:

stack new no-annex --resolver https://gist.github.com/dbaynard/9693107a883a6282f52f22785d0072e2/raw/edd32d61464cf0e19236081e250aebbe8cea1112

The resolver includes a git repository which uses git-annex.

Expected

Stack to create the directory and populate with a stack.yaml file.

Actual

Stack creates the directory, then attempts to resolve the git repos in the snapshot, fails with https://github.com/dbaynard/cascade, and so doesn't create the stack.yaml file. The command succeeds on 1.9.3.

Downloading template "new-template" to create project "no-annex" in no-annex/ ...
Looking for .cabal or package.yaml files to use to init the project.
Using cabal packages:
- no-annex/

Selected resolver: 8.6.3-13.5-baynard-snapshot
Loading a snapshot from a SnapshotDef: 8.6.3-13.5-baynard-snapshot
Loading a snapshot from a SnapshotDef: lts-13.5
Loading a snapshot from a SnapshotDef: ghc-8.6.3
Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
Unsupported tarball from /tmp/get-repo18829/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

Verbose log

Version 1.10.0 (6902 commits) x86_64 hpack-0.31.1
2019-02-10 17:30:20.358305: [debug] Opening local template: "new-template.hsfiles"
2019-02-10 17:30:20.358606: [info] Downloading template "new-template" to create project "no-annex" in no-annex/ ...
2019-02-10 17:30:20.358685: [debug] Downloading /commercialhaskell/stack-templates/master/new-template.hsfiles
2019-02-10 17:30:20.795431: [debug] Opening local template: "/home/db/.stack/templates/new-template.hsfiles"
2019-02-10 17:30:20.809348: [info] Looking for .cabal or package.yaml files to use to init the project.
2019-02-10 17:30:20.811577: [info] Using cabal packages:
2019-02-10 17:30:20.812241: [info] - no-annex/

2019-02-10 17:30:20.812622: [debug] Running hpack on /home/db/Projects/stack-triage/no-annex/package.yaml
2019-02-10 17:30:20.816604: [debug] hpack generated a modified version of /home/db/Projects/stack-triage/no-annex/no-annex.cabal
2019-02-10 17:30:20.967687: [info] Selected resolver: 8.6.3-13.5-baynard-snapshot
2019-02-10 17:30:20.970425: [debug] Trying to decode /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache
2019-02-10 17:30:20.970564: [debug] Exception ignored when attempting to load /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache: /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache: openBinaryFile: does not exist (No such file or directory)
2019-02-10 17:30:20.970653: [debug] Failure decoding /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache
2019-02-10 17:30:20.970713: [info] Loading a snapshot from a SnapshotDef: 8.6.3-13.5-baynard-snapshot
2019-02-10 17:30:20.970814: [debug] Trying to decode /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/d7f31aacdd7580839a49a29174d1025edd557547229c8e84bdc74bf8aee94c56.cache
2019-02-10 17:30:21.103588: [debug] Success decoding /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/d7f31aacdd7580839a49a29174d1025edd557547229c8e84bdc74bf8aee94c56.cache
2019-02-10 17:30:21.103745: [debug] Parsing cabal file for Repo from https://github.com/jameysharp/corrode, commit 34053342c2f1ca04f23ad94d67057f14e74d9fb9
2019-02-10 17:30:21.106157: [debug] Parsing cabal file for Repo from https://github.com/mgattozzi/curryrs, commit 6d3ecbc3d1786ccc16e2ff20c6d6da53734f6b03
2019-02-10 17:30:21.108578: [debug] Parsing cabal file for Repo from https://github.com/smallhadroncollider/taskell, commit 3c2b5bb272df616670202bb3da42e1dc0606015a
2019-02-10 17:30:21.114289: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict
2019-02-10 17:30:21.118805: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-db
2019-02-10 17:30:21.120918: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-html
2019-02-10 17:30:21.122066: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-json
2019-02-10 17:30:21.123896: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-quickcheck
2019-02-10 17:30:21.125179: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay
2019-02-10 17:30:21.127342: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay-data
2019-02-10 17:30:21.129108: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay-serial
2019-02-10 17:30:21.130675: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir readp
2019-02-10 17:30:21.132285: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/hserv, commit 736555ca4869da6d8b3d818df5d32a0831a58488
2019-02-10 17:30:21.133937: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/htidy, commit a4139a34f128a858169218211565cae7e1ba5752
2019-02-10 17:30:21.135793: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/pandoc-include, commit ad1f72c485c4c00908326a90efe94ead9bb73c77
2019-02-10 17:30:21.137710: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/lucid-from-html, commit 901bb8edf5e77f227cde847763ee429c620ef62b in subdir lucid-from-html
2019-02-10 17:30:21.140073: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/cascade, commit 9699b1a6f61c28ce154f271aef9bb784f65fd47e in subdir cascade
2019-02-10 17:30:21.140958: [info] Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
2019-02-10 17:30:21.141851: [debug] Run process within /tmp/get-repo19260: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-10 17:30:22.705593: [debug] Process finished in 1563ms: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-10 17:30:22.705788: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-10 17:30:22.714857: [debug] Process finished in 9ms: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-10 17:30:22.715085: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git submodule update --init --recursive
2019-02-10 17:30:22.785615: [debug] Process finished in 70ms: /usr/bin/git submodule update --init --recursive
2019-02-10 17:30:22.785991: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo19260/foo.tar HEAD
2019-02-10 17:30:22.798792: [debug] Process finished in 12ms: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo19260/foo.tar HEAD
2019-02-10 17:30:22.802218: [debug] parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo19260/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

Stack version

$ stack --version
Version 1.10.0 (6902 commits) x86_64 hpack-0.31.1

I've no idea why it doesn't show the git ref.

Method of installation

Built from git repo.

The text was updated successfully, but these errors were encountered:

dbaynard · 2019-02-10T17:37:12Z

Note that if the directory already exists, stack build fails for the same reason.

dbaynard · 2019-02-10T17:40:57Z

I'd like stack to support git-annex, but I'm happy to consider it out of scope. This error ought to be handled, though. Would you mind taking a look, @qrilka?

dbaynard · 2019-02-10T20:56:54Z

Actually, in this case I don't need any git-annex files to build the package so it ought to work.

qrilka · 2019-02-10T21:04:02Z

Sure, I will try to find time to investigate this

qrilka · 2019-02-12T21:18:20Z

@dbaynard this doesn't appear to be related to tar problems, e.g. I see the following verbose output:

2019-02-13 00:02:05.712578: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/cascade, commit 9699b1a6f61c28ce154f271aef9bb784f65fd47e in subdir cascade
2019-02-13 00:02:05.712919: [info] Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
2019-02-13 00:02:05.713000: [debug] Run process within /tmp/get-repo10938: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-13 00:02:07.521022: [debug] Process finished in 1808ms: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-13 00:02:07.521206: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-13 00:02:07.526820: [debug] Process finished in 6ms: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-13 00:02:07.526981: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git submodule update --init --recursive
2019-02-13 00:02:07.554099: [debug] Process finished in 27ms: /usr/bin/git submodule update --init --recursive
2019-02-13 00:02:07.554269: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo10938/foo.tar HEAD
2019-02-13 00:02:07.557223: [debug] Process finished in 3ms: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo10938/foo.tar HEAD
2019-02-13 00:02:07.558883: [debug] parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo10938/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

See git archive being used - that's the way Pantry currently deals with Git repos - it just gets an archive and uses archive handling code and as your archive contains annex files and Pantry doesn't do anything about them you get broken links in tar file and Pantry fails.
@snoyberg could you give more details why we go through archives for Git repos?

snoyberg · 2019-02-13T03:46:40Z

Yes:

It can be more efficient than asking git to populate the filesystem
On filesystems without proper symbolic link support or case insensitivity, it's the only way to ensure we get reproducible repo contents, otherwise the tree hashes would be different on different OSes

We could theoretically make Pantry simply ignore broken symlinks, but the error we're getting here is due to more rigorous error checking.

qrilka · 2019-02-13T12:36:26Z

@dbaynard the only solution I could propose in this case is to add fonts export-ignore to .gitattributes of your git-annex repo. If that sounds as a good workaround then it makes sense to document it.

dbaynard · 2019-02-13T15:54:57Z

Yes, fixing the archive seems like the right way to do this. I'll try your suggestion. I wonder if there are other cases where archiving fails, too?

Meanwhile:

parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo10938/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

This snippet shows the code which is throwing this error (the upper case is the end of the definition of toSimple).

stack/subs/pantry/src/Pantry/Archive.hs

Lines 384 to 393 in 7a076ab

    
                     case Map.lookup dest files of 
        
                       Nothing -> Left $ "Symbolic link dest not found from " ++ mePath me ++ " to " ++ relDest ++ ", looking for " ++ dest 
        
                       Just me' -> 
        
                         case meType me' of 
        
                           METNormal -> Right $ SimpleEntry dest FTNormal 
        
                           METExecutable -> Right $ SimpleEntry dest FTExecutable 
        
                           METLink _ -> Left $ "Symbolic link dest cannot be a symbolic link, from " ++ mePath me ++ " to " ++ relDest 
        
           case traverse toSimple files of 
        
             Left e -> throwIO $ UnsupportedTarball loc $ T.pack e

Would line 385 be a good place to add that stack doesn't support git-annex, and provide a link to the workaround?

qrilka · 2019-02-19T15:20:57Z

@dbaynard that code deals with archives in general so I'm not sure we could talk explicitly about git-annex here, probably we could give some hint that it could be the reason behind such a failure but also there are other options - e.g. archive could come not from git at all and bad symlinks could get put into it my some accident.

qrilka · 2019-02-28T14:15:58Z

@dbaynard what do you think could be a proper resolution for this issue?

dbaynard · 2019-03-03T16:35:37Z

give some hint that it could be the reason behind such a failure but also there are other options - e.g. archive could come not from git at all and bad symlinks could get put into it my some accident.

This sounds like an improvement.

See commercialhaskell/stack#4579

dbaynard · 2019-03-07T18:41:40Z

the only solution I could propose in this case is to add fonts export-ignore to .gitattributes of your git-annex repo.

This works. Thanks @qrilka! I'll PR with some documentation.

running into issues building hx-deploy-tool , according to the error message, this fixes the problem commercialhaskell/stack#4579

Latest stack doesn't seem to support named commits such as 'master'. Get the commit sha from `git rev-parse` and append it to the yaml file. This doesn't fix google#391 but goes one step further. We still get an error like: $ stack runghc -- bootstrap.hs Cloning 0bef8c2 from https://github.com/google/proto-lens Unsupported tarball from /tmp/with-repo-archive189930/foo.tar: Symbolic link dest not found from proto-lens/proto-lens-imports/google to ../../google/protobuf/src/google, looking for proto-lens/../google/protobuf/src/google. This may indicate that the source is a git archive which uses git-annex. See commercialhaskell/stack#4579 for further information. bootstrap.hs: readCreateProcess: stack "--stack-yaml=stack-bootstrap.yaml" "path" "--local-install-root" (exit 1): failed

Latest stack doesn't seem to support named commits such as 'master'. Get the commit sha from `git rev-parse` and append it to the yaml file. This doesn't fix #391 but goes one step further. We still get an error like: $ stack runghc -- bootstrap.hs Cloning 0bef8c2 from https://github.com/google/proto-lens Unsupported tarball from /tmp/with-repo-archive189930/foo.tar: Symbolic link dest not found from proto-lens/proto-lens-imports/google to ../../google/protobuf/src/google, looking for proto-lens/../google/protobuf/src/google. This may indicate that the source is a git archive which uses git-annex. See commercialhaskell/stack#4579 for further information. bootstrap.hs: readCreateProcess: stack "--stack-yaml=stack-bootstrap.yaml" "path" "--local-install-root" (exit 1): failed

…stack` deleted dead symlink "cddl-spec" The presence of an invalid link leads to an error in the `cardano-node` build via `stack`. See commercialhaskell/stack#4579 for further information.

Right now the following `stack.yaml`: ``` extra-deps: - git: https://github.com/hspec/hspec commit: c38a03d ``` Yields: ``` Cloning c38a03d from https://github.com/hspec/hspec Error: [S-760] Unsupported tarball from /dev/shm/drlkf/with-repo-archive2139961/foo.tar: Symbolic link dest not found from hspec-meta/vendor to hspec-core/vendor/, looking for hspec-meta/hspec-core/vendor. This may indicate that the source is a git archive which uses git-annex. See commercialhaskell/stack#4579 for further information. ``` See also: commercialhaskell/stack#5004

dbaynard added type: bug component: pantry labels Feb 10, 2019

dbaynard assigned qrilka Feb 10, 2019

This was referenced Feb 11, 2019

Can't use Stack's github as an extra-dep: Tar error #4580

Closed

Add support for Git repositories with submodules #4581

Merged

dbaynard added a commit to dbaynard/cascade that referenced this issue Mar 7, 2019

Add fonts to export-ignore

afe8fd5

See commercialhaskell/stack#4579

dbaynard added the awaiting pull request label Mar 9, 2019

dbaynard self-assigned this Mar 9, 2019

dbaynard mentioned this issue Mar 9, 2019

Indicate where git-annex will lead to pantry error #4615

Merged

2 tasks

snoyberg closed this as completed in #4615 Mar 26, 2019

ashvds added a commit to ashvds/adl that referenced this issue Jul 4, 2019

Adding 'fonts export-ignore' as per haskel doco

7782aff

running into issues building hx-deploy-tool , according to the error message, this fixes the problem commercialhaskell/stack#4579

This was referenced Jul 4, 2019

Adding 'fonts export-ignore' as per haskel doco adl-lang/adl#70

Closed

ADL import fails on stack build helix-collective/hx-deploy-tool#34

Closed

mattaudesse mentioned this issue Aug 13, 2019

Git dependencies fail when they contain symlinks to outside the repository, or broken symlinks #4913

Open

jinwoo mentioned this issue Jun 3, 2020

Use commit sha in stack-bootstrap.yaml google/proto-lens#392

Merged

jinwoo mentioned this issue Jun 3, 2020

bootstrap.hs doesn't work with stack 2.3.1 (Cannot complete repo information for a non SHA1 commit due to non-reproducibility) google/proto-lens#391

Open

Fell-x27 mentioned this issue Aug 24, 2020

Can't build: Unsupported tarball from /tmp/with-repo-archive4121/foo.tar cardano-foundation/cardano-wallet#2060

Closed

brandon-leapyear mentioned this issue Aug 20, 2021

Can't add repo to extra-deps if repo contains symlink to git submodule #5603

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pantry does not work with git annex #4579

Pantry does not work with git annex #4579

dbaynard commented Feb 10, 2019 •

edited

Loading

dbaynard commented Feb 10, 2019 •

edited

Loading

dbaynard commented Feb 10, 2019

dbaynard commented Feb 10, 2019

qrilka commented Feb 10, 2019

qrilka commented Feb 12, 2019

snoyberg commented Feb 13, 2019

qrilka commented Feb 13, 2019

dbaynard commented Feb 13, 2019

qrilka commented Feb 19, 2019

qrilka commented Feb 28, 2019

dbaynard commented Mar 3, 2019

dbaynard commented Mar 7, 2019

Pantry does not work with git annex #4579

Pantry does not work with git annex #4579

Comments

dbaynard commented Feb 10, 2019 • edited Loading

General summary/comments (optional)

Steps to reproduce

Expected

Actual

Stack version

Method of installation

dbaynard commented Feb 10, 2019 • edited Loading

dbaynard commented Feb 10, 2019

dbaynard commented Feb 10, 2019

qrilka commented Feb 10, 2019

qrilka commented Feb 12, 2019

snoyberg commented Feb 13, 2019

qrilka commented Feb 13, 2019

dbaynard commented Feb 13, 2019

qrilka commented Feb 19, 2019

qrilka commented Feb 28, 2019

dbaynard commented Mar 3, 2019

dbaynard commented Mar 7, 2019

dbaynard commented Feb 10, 2019 •

edited

Loading

dbaynard commented Feb 10, 2019 •

edited

Loading