Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal libraries break stack haddock #3989

Closed
ryanreich opened this issue Apr 27, 2018 · 45 comments · Fixed by #4596
Closed

Internal libraries break stack haddock #3989

ryanreich opened this issue Apr 27, 2018 · 45 comments · Fixed by #4596

Comments

@ryanreich
Copy link

The issue reported in haskell/cabal#4969 appears not to be fixed in stack: building haddock documentation for a package with internal libraries causes an error while resolving dependencies of the main library. The resolution proposed there, to use cabal new-haddock, has no obvious stack equivalent.

@istathar
Copy link
Contributor

istathar commented May 1, 2018

I've been following links to issues between projects and the long chain eventually led me to #3955 which is merged. But this issue is still occurring (as of lts-11.7 and stack-1.7.1) so I'm at a bit of a loss.

    Preprocessing library for haddock-library-1.4.5..
    Running Haddock on library for haddock-library-1.4.5..
    Cabal-simple_mPHDZzAJ_2.0.1.0_ghc-8.2.2: internal error when calculating
    transitive package dependencies.
    Debug info: []

@mihaimaruseac mihaimaruseac self-assigned this May 2, 2018
@mihaimaruseac
Copy link
Contributor

Thanks for the report, I was able to reproduce and I'll try to solve it (I think it's close to a solution for #3996)

@istathar
Copy link
Contributor

istathar commented May 3, 2018

@mihaimaruseac Trying to build pandoc, for what it's worth. I'm sure there's a simpler minimal reproducible case.

@mihaimaruseac
Copy link
Contributor

mihaimaruseac commented May 3, 2018

There is, I reproduced it from the example in #3996 (although there is no documentation there to be generated)

@istathar
Copy link
Contributor

Does anyone know of a workaround? Should revert to stack-1.6? I'm quite unable to build my projects right now because of this.

@mihaimaruseac
Copy link
Contributor

Sorry, I stopped working on it to get HCAR out. I'll restart working on this after Tuesday/Wednesday when i hope to get HCAR public.

@istathar
Copy link
Contributor

@mihaimaruseac certainly. Most appreciated. I'm just a bit surprised that I couldn't find an obvious way to get around this. Usually in stack if something goes wrong it's worst-case nuke the .stack-work directory. :). Look forward to trying any code you want me to test if/when the time comes.

@NorfairKing
Copy link
Contributor

I just bumped into this problem as well. I haven't found a workaround.

@NorfairKing
Copy link
Contributor

@mihaimaruseac, any luck?

@mihaimaruseac
Copy link
Contributor

Not yet, still working on it.

@mihaimaruseac
Copy link
Contributor

I made #4033 for #3966, next step is this. With the repo to reproduce #3966 I have a successful stack haddock if the internal library is a dependency of the main library but not if not.

@mihaimaruseac
Copy link
Contributor

@afcowie, do you recall which version of Pandoc exercised this bug? I tried it now with the latest but it no longer has internal libraries so it works for me.

@mihaimaruseac
Copy link
Contributor

I found a workaround: with the fix in #4033 and if the internal library is not a dependency of the main library, stack haddock finishes successfully.

I'll keep looking into this, but so far it seems that both in the case when the sublib is a dependency of the main library and not, Stack calls cabal haddock with the exact same arguments, after properly building the code. Surely I'm missing something.

@istathar
Copy link
Contributor

istathar commented May 25, 2018

do you recall which version of Pandoc exercised this bug?

The one in resolver lts-12.20. I just tried still happening six months later

$ stack haddock pandoc

with lts-12.20 and no change. Which resolver are you using, @mihaimaruseac?

@mihaimaruseac
Copy link
Contributor

Oh, I stack unpack'ed the latest version and tried building it directly.

@mihaimaruseac
Copy link
Contributor

Related issue pointing to why the Stackage curator doesn't run haddock on haddock-library. commercialhaskell/stackage#3236

Seems to be a bug in Cabal, not in Stack. I'll be trying the rest of today to get a workaround, but I don't know how many chances of success I have.

@mihaimaruseac
Copy link
Contributor

Tried making Stack run the steps suggested by haskell/cabal#4969 (comment) but it fails due to GHC_PACKAGE_PATH.

I'm going to put this on the back of my mind and work on something else for a while and maybe I'll get more ideas in meanwhile. Or maybe cabal gets updated to make the new-build commands the default and then Stack can pick it up and adapt.

Sorry for not being able to solve this :(

@mihaimaruseac mihaimaruseac removed their assignment May 29, 2018
@ip1981
Copy link

ip1981 commented Jun 20, 2018

I think this could be fixed by linking the Setup program with the recent Cabal library (2.3.0.0+), but can't find any way to do it with Stack.

@ip1981
Copy link

ip1981 commented Jun 20, 2018

Or by disabling haddock for some packages.

@dbaynard
Copy link
Contributor

@mihaimaruseac should #4111 fix this?

@mihaimaruseac
Copy link
Contributor

I don't think so. I'm going to test it later today or tomorrow, but I'm almost confident it doesn't as this is blocked on calling Cabal in a totally different way than Stack does.

@mihaimaruseac
Copy link
Contributor

Checked now, this is still an issue.

geigerzaehler pushed a commit to oscoin/oscoin that referenced this issue Nov 5, 2018
Also changes the repo to Github since syncing to the `git.oscoin.io`
mirror is unreliable.

We need to disable building the haddocks temporarily because of
commercialhaskell/stack#3989
@istathar
Copy link
Contributor

Is it possible to selectively disable the generation of haddocks on failing librairies to circumvent this issue ?

More specifically, what file do I need to touch (ie create zero size) so that stack doesn't think that it needs to build haddock content for haddock-library. I'd be happy to use that as a workaround for now.

AfC

@mihaimaruseac
Copy link
Contributor

I don't think it's Stack. Stack just calls Setup.hs to build the haddocks, so probably it's a question for Cabal/GHC

@qrilka
Copy link
Contributor

qrilka commented Dec 27, 2018

It looks like this Cabal problem was fixed in version 2.4.0.1, at least I got stack haddock succeeding with test/integration/tests/3899-dont-rebuild-sublibraries/files by switching from 8.2.2 to 8.6.3, I plan to try haddock-library a bit later

@istathar
Copy link
Contributor

For what it's worth, pandoc 2.5 depends on haddock-library 1.7.0 which has dropped the troublesome internal (vendored?) dependency on attoparsec. So the problem is going to go away for people like me when that version hits Stackage. Which is great for me, but soon I won't have a clear idea if the underlying problem is fixed. I hope @qrilka's report is correct.

@qrilka
Copy link
Contributor

qrilka commented Dec 27, 2018

stack haddock works fine for haddock-library with LTS-13.0

@mihaimaruseac
Copy link
Contributor

The underlying problem most likely isn't fixed, it's just that haddock-libary devs were convinced to not use internal deps.

Didn't get a chance to investigate the underlying problem yet, I'm too far back on open source work :(

@qrilka
Copy link
Contributor

qrilka commented Dec 28, 2018

@mihaimaruseac why do you think it's not fixed?
Here is a log of successful build:

qrilka@qdesktop ~/ws/h/haddock $ git checkout ghc-8.4
Ветка «ghc-8.4» отслеживает внешнюю ветку «ghc-8.4» из «origin».
Переключено на новую ветку «ghc-8.4»
qrilka@qdesktop ~/ws/h/haddock $ PAGER= git diff haddock-library
diff --git a/haddock-library/haddock-library.cabal b/haddock-library/haddock-library.cabal
index eb961041..e6819f79 100644
--- a/haddock-library/haddock-library.cabal
+++ b/haddock-library/haddock-library.cabal
@@ -22,9 +22,9 @@ library
   default-language:     Haskell2010
 
   build-depends:
-      base         >= 4.5     && < 4.12
+      base         >= 4.5     && < 4.13
     , bytestring   >= 0.9.2.1 && < 0.11
-    , containers   >= 0.4.2.1 && < 0.6
+    , containers   >= 0.4.2.1 && < 0.7
     , transformers >= 0.3.0   && < 0.6
     , text         >= 1.2.3.0  && < 1.3
     , parsec       >= 3.1.13.0 && < 3.2
qrilka@qdesktop ~/ws/h/haddock $ grep ^version haddock-library/haddock-library.cabal 
version:              1.6.0
qrilka@qdesktop ~/ws/h/haddock $ stack haddock
haddock-library-1.6.0: unregistering (rebuilding with haddocks)
haddock-library-1.6.0: configure (lib)
Configuring haddock-library-1.6.0...
haddock-library-1.6.0: build (lib)
Preprocessing library for haddock-library-1.6.0..
Building library for haddock-library-1.6.0..
haddock-library-1.6.0: haddock
Preprocessing library for haddock-library-1.6.0..
Running Haddock on library for haddock-library-1.6.0..
Haddock coverage:
Warning: 'Doc' is out of scope.
    If you qualify the identifier, haddock can try to link it
    it anyway.
  19% (  3 / 16) in 'Documentation.Haddock.Types'
  Missing documentation for:
    MetaDoc (src/Documentation/Haddock/Types.hs:41)
    overDoc (src/Documentation/Haddock/Types.hs:59)
    overDocF (src/Documentation/Haddock/Types.hs:62)
    Version (src/Documentation/Haddock/Types.hs:65)
    Package (src/Documentation/Haddock/Types.hs:66)
    Hyperlink (src/Documentation/Haddock/Types.hs:68)
    Picture (src/Documentation/Haddock/Types.hs:73)
    Header (src/Documentation/Haddock/Types.hs:78)
    Example (src/Documentation/Haddock/Types.hs:83)
    TableCell (src/Documentation/Haddock/Types.hs:88)
    TableRow (src/Documentation/Haddock/Types.hs:94)
    Table (src/Documentation/Haddock/Types.hs:98)
    DocH (src/Documentation/Haddock/Types.hs:103)
  57% (  8 / 14) in 'Documentation.Haddock.Parser.Monad'
  Missing documentation for:
    Module header
    ParserState (src/Documentation/Haddock/Parser/Monad.hs:23)
    initialParserState (src/Documentation/Haddock/Parser/Monad.hs:27)
    setSince (src/Documentation/Haddock/Parser/Monad.hs:30)
    Parser (src/Documentation/Haddock/Parser/Monad.hs:33)
    parseOnly (src/Documentation/Haddock/Parser/Monad.hs:38)
  67% (  2 /  3) in 'Documentation.Haddock.Markup'
  Missing documentation for:
    markup (src/Documentation/Haddock/Markup.hs:9)
  44% (  4 /  9) in 'Documentation.Haddock.Doc'
  Missing documentation for:
    Module header
    docParagraph (src/Documentation/Haddock/Doc.hs:57)
    docAppend (src/Documentation/Haddock/Doc.hs:41)
    docConcat (src/Documentation/Haddock/Doc.hs:11)
    emptyMetaDoc (src/Documentation/Haddock/Doc.hs:35)
Warning: '#' is out of scope.
    If you qualify the identifier, haddock can try to link it
    it anyway.
 100% (  6 /  6) in 'Documentation.Haddock.Parser'
  67% (  2 /  3) in 'Documentation.Haddock.Utf8'
  Missing documentation for:
    Module header
Warning: Documentation.Haddock.Parser: could not find link destinations for:
    parseText parseParagraph innerList definitionList more parseValid
Warning: Documentation.Haddock.Utf8: could not find link destinations for:
    encode decode
Documentation created:
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/index.html,
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/haddock-library.txt
haddock-library-1.6.0: copy/register
Installing library in /home/qrilka/ws/h/haddock/.stack-work/install/x86_64-linux-tinfo6/lts-13.0/8.6.3/lib/x86_64-linux-ghc-8.6.3/haddock-library-1.6.0-IoJUVQoM6qZAhuN6fYIkNo
Registering library for haddock-library-1.6.0..
Updating Haddock index for local packages in
/home/qrilka/ws/h/haddock/.stack-work/install/x86_64-linux-tinfo6/lts-13.0/8.6.3/doc/index.html
Updating Haddock index for local packages and dependencies in
/home/qrilka/ws/h/haddock/.stack-work/install/x86_64-linux-tinfo6/lts-13.0/8.6.3/doc/all/index.html
Haddock index for snapshot packages already up to date at:
/home/qrilka/.stack/snapshots/x86_64-linux-tinfo6/lts-13.0/8.6.3/doc/index.html
qrilka@qdesktop ~/ws/h/haddock $ 

@mihaimaruseac
Copy link
Contributor

Try haddock-library 1.4.5

@qrilka
Copy link
Contributor

qrilka commented Dec 28, 2018

@mihaimaruseac
Copy link
Contributor

No, it doesn't: https://gist.github.com/mihaimaruseac/8eb45ef4ba223c6c801c8cd6d99000e8

You might have some env or binary in path which made it work, but it doesn't on a fresh install of HEAD of Stack.

@qrilka
Copy link
Contributor

qrilka commented Dec 29, 2018

I see "ghc-8.2.2" in you gist @mihaimaruseac - how do you get it with LTS-13.0? I guess that's why you have that problem. BTW I still have old stack-1.7.1 on by $PATH by default, the problem isn't about Stack, it's about the change in how Cabal works.

@mihaimaruseac
Copy link
Contributor

Selected resolver: lts-11.22

I was under the impression that this is a Cabal internals issue that Stack needs a work around for.

@qrilka
Copy link
Contributor

qrilka commented Dec 29, 2018

Oh, missed that line, sorry, from a couple of Cabal issues I've read it looks like Stack could have some workaround even with some previous Cabal versions (though I'm not sure how) but it appears that now we don't need to :)

@mihaimaruseac
Copy link
Contributor

mihaimaruseac commented Dec 29, 2018

Even manually switching to the LTR-13.0 resolver doesn't work

[haddock-library-1.4.5] λ cat stack.yaml 
resolver: lts-13.0
packages:
- .
allow-newer: true
[haddock-library-1.4.5] λ stack haddock
WARNING: Ignoring out of range dependency (allow-newer enabled): base-4.12.0.0. haddock-library requires: >=4.5 && <4.11
haddock-library-1.4.5: configure (lib + internal-lib)
Configuring haddock-library-1.4.5...
haddock-library-1.4.5: build (lib + internal-lib)
Preprocessing library 'attoparsec' for haddock-library-1.4.5..
Building library 'attoparsec' for haddock-library-1.4.5..
[ 1 of 11] Compiling Data.Attoparsec.ByteString.FastSet ( vendor/attoparsec-0.13.1.0/Data/Attoparsec/ByteString/FastSet.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/attoparsec/Data/Attoparsec/ByteString/FastSet.o )
...
[11 of 11] Compiling Data.Attoparsec.Number ( vendor/attoparsec-0.13.1.0/Data/Attoparsec/Number.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/attoparsec/Data/Attoparsec/Number.o )
Preprocessing library for haddock-library-1.4.5..
Building library for haddock-library-1.4.5..
[1 of 7] Compiling Documentation.Haddock.Types ( src/Documentation/Haddock/Types.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/Documentation/Haddock/Types.o )
...
[7 of 7] Compiling Documentation.Haddock.Parser ( src/Documentation/Haddock/Parser.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/Documentation/Haddock/Parser.o )
haddock-library-1.4.5: haddock
Preprocessing library 'attoparsec' for haddock-library-1.4.5..
Running Haddock on library 'attoparsec' for haddock-library-1.4.5..
Haddock coverage:
...
Documentation created:
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/index.html,
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/haddock-library.txt
Preprocessing library for haddock-library-1.4.5..
Running Haddock on library for haddock-library-1.4.5..
Haddock coverage:
...
Documentation created:
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/index.html,
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/haddock-library.txt
haddock-library-1.4.5: copy/register
Installing internal library attoparsec in /tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/lib/x86_64-linux-ghc-8.6.3/haddock-library-1.4.5-4aIdHtThdzDHvICUM0c4eh-attoparsec
Installing library in /tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/lib/x86_64-linux-ghc-8.6.3/haddock-library-1.4.5-62JBiw4RYQM30IALTFPWTm
Registering library 'attoparsec' for haddock-library-1.4.5..
Registering library for haddock-library-1.4.5..
Updating Haddock index for local packages in
/tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/doc/index.html
Updating Haddock index for local packages and dependencies in
/tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/doc/all/index.html
Received ExitFailure 1 when running
Raw command: /home/mihai/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/haddock --optghc=-package-db=/home/mihai/.stack/snapshots/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/pkgdb --optghc=-package-db=/tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/pkgdb --gen-contents --gen-index -i ../array-0.5.3.0,../array-0.5.3.0/array.haddock -i ../ghc-prim-0.5.3,../ghc-prim-0.5.3/ghc-prim.haddock -i ../bytestring-0.10.8.2,../bytestring-0.10.8.2/bytestring.haddock -i ../z-haddock-library-z-attoparsec-1.4.5,../z-haddock-library-z-attoparsec-1.4.5/z-haddock-library-z-attoparsec.haddock -i ../transformers-0.5.5.0,../transformers-0.5.5.0/transformers.haddock -i ../haddock-library-1.4.5,../haddock-library-1.4.5/haddock-library.haddock -i ../base-4.12.0.0,../base-4.12.0.0/base.haddock -i ../deepseq-1.4.4.0,../deepseq-1.4.4.0/deepseq.haddock -i ../integer-gmp-1.0.2.0,../integer-gmp-1.0.2.0/integer-gmp.haddock
Run from: /tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/doc/all/
Standard output:

haddock: internal error: ../z-haddock-library-z-attoparsec-1.4.5/z-haddock-library-z-attoparsec.haddock: openBinaryFile: does not exist (No such file or directory)

@qrilka
Copy link
Contributor

qrilka commented Dec 29, 2018

@mihaimaruseac I think I have found was causes this for you: see special hack which was added exactly for this library - https://github.com/commercialhaskell/stack/blob/v1.9.0.1/src/Stack/Build/Execute.hs#L1328
I have stack-1.7.1 locally so I don't have that enabled and so my Stack works OK, I guess that thing needs get reviewed.

@mihaimaruseac
Copy link
Contributor

I added that, in #4111 as it was solving other issues with internal libraries.

@mihaimaruseac
Copy link
Contributor

Oh, but it looks like the error changed from the beginning of the issue, so maybe there's some chance of fixing this. If I get some time I'll investigate.

@mihaimaruseac
Copy link
Contributor

One final point as to why this is not solved yet: I got a container with stahck-1.7.1 and, while stack haddock succeeds, no documentation for the internal library is generated (it is ignored completely).

@qrilka
Copy link
Contributor

qrilka commented Dec 31, 2018

I completely forgot about the docs themselves. I see docs for the internal library getting generated:

qrilka@qdesktop ~/ws/h/haddock-library-1.4.5 $ ls `stack path --local-doc-root`/haddock-library-1.4.5
Data-Attoparsec-ByteString-Char8.html    Documentation-Haddock-Parser.html  haddock-bundle.min.js    meta.json       src
Data-Attoparsec-ByteString.html          Documentation-Haddock-Types.html   haddock-library.haddock  minus.gif       synopsis.png
Documentation-Haddock-Doc.html           Documentation-Haddock-Utf8.html    haddock-library.txt      ocean.css
Documentation-Haddock-Markup.html        LICENSE                            hslogo-16.png            plus.gif
Documentation-Haddock-Parser-Monad.html  doc-index.html                     index.html               quick-jump.css

But it looks like main library index doesn't include them (it overwrites the one from the internal library).
I tried cabal new-haddock (but with Cabal 2.2) - it generates 2 haddock directories and the one about internal attoparsec talks about "haddock-library-1.4.5: Library exposing some functionality of Haddock." which is at least very confusing...

@jarcane
Copy link

jarcane commented May 14, 2019

I am still experiencing this issue. We have a large monorepo that still fails stack build --haddock on 1.9.3.1 because of an internal library in one of the submodules.

I get: haddock: internal error: ../z-avatar-app-z-avatar-internal-0/z-avatar-app-z-avatar-internal.haddock: openBinaryFile: does not exist (No such file or directory) when trying to do the haddock build in root.

You can find the offending .cabal file here, if that's helpful: https://github.com/futurice/haskell-mega-repo/blob/master/avatar-app/avatar-app.cabal
and stack.yaml: https://github.com/futurice/haskell-mega-repo/blob/master/stack.yaml

Any idea when the fix in #4596 will be released?

@snoyberg
Copy link
Contributor

That change will be released with Stack 2. You can see the blog post for more details, but the earliest release date is May 29. In the meanwhile, you can try using the prerelease version. You can try stack upgrade --git, or watch for an announcement in the next few days about a release candidate.

@jarcane
Copy link

jarcane commented May 15, 2019

Thank you. I was able to build correctly on the Git build of Stack, so looks like we'll just have to wait until the update.

dcastro added a commit to dcastro/haskell-flatbuffers that referenced this issue Jun 11, 2019
There's a bug in stack 1.9.x and `stack haddock` breaks when there's an internal library.
For now, we're trying a newer version of stack where this bug should be fixed.

commercialhaskell/stack#3989
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment