Stack dot should handle properly GHC boot packages

[qrilka]

General summary/comments (optional)

It looks like currently stack dot doesn't show dependencies of GHC boot libraries even when such a library comes from a Stackage LTS snapshot. Either the code should be improved by including those dependencies or it should be documented that stack doesn't track dependencies of GHC boot packages.

Steps to reproduce

In the root of the stack repository run stack dot --external | grep '"process"'

Expected

Dependencies of the process package will get shown (e.g. directory or filepath) besides the packages which depend on process.

Actual

Current output (both for versions 1.7.1 and prerelese 1.9.0.1):

{rank=max; "process"; };
"Cabal" -> "process";
"cmdargs" -> "process";
"conduit-extra" -> "process";
"echo" -> "process";
"extra" -> "process";
"githash" -> "process";
"gitrev" -> "process";
"open-browser" -> "process";
"optparse-applicative" -> "process";
"rio" -> "process";
"stack" -> "process";
"streaming-commons" -> "process";
"typed-process" -> "process";
"unliftio" -> "process";
"x509-system" -> "process";

Stack version

$ stack --numeric-version
1.7.1
$ ~/bin/stack-1.9.0.1 --numeric-version
1.9.0.1

Method of installation

• Other : 1.7.1 install with gentoo-haskell, prerelease version downloaded from the Github release page

[dbaynard]

I thought --include-base might help but it doesn't.

This isn't totally obvious to me — why is it valuable to display boot libraries?

We may want to add an option --include-boot, or modify --include-base, or make it default, depending on the answer.

[qrilka]

Omitting some dependencies results in an incomplete dependency graph, I think Stack should be explicit about this. And actually we could add this information e.g. by using ghc-pkg describe <pkg>

[vanceism7]

Hey guys, I've been looking at this issue and I think I was able to get those dependencies to load up in the dot graph.

So here's the offending code thats causing certain dependencies to not show.
In Dot.hs we have the function createDepLoader like so:

createDepLoader sourceMap installed globalDumpMap globalIdMap loadPackageDeps pkgName =
  if not (pkgName `Set.member` wiredInPackages)
      then case Map.lookup pkgName (smProject sourceMap) of
          Just pp -> do
            pkg <- loadCommonPackage (ppCommon pp)
            pure (packageAllDeps pkg, payloadFromLocal pkg)
          Nothing ->
            case Map.lookup pkgName (smDeps sourceMap) of
              Just DepPackage{dpLocation=PLMutable dir} -> do
                pp <- mkProjectPackage YesPrintWarnings dir False
                pkg <- loadCommonPackage (ppCommon pp)
                pure (packageAllDeps pkg, payloadFromLocal pkg)
              Just dp@DepPackage{dpLocation=PLImmutable loc} -> do
                let common = dpCommon dp
                gpd <- liftIO $ cpGPD common
                let PackageIdentifier name version = PD.package $ PD.packageDescription gpd
                    flags = cpFlags common
                    ghcOptions = cpGhcOptions common
                assert (pkgName == name) (loadPackageDeps pkgName version loc flags ghcOptions)
              Nothing ->
                pure (Set.empty, payloadFromInstalled (Map.lookup pkgName installed))

That last bit of code at the end is the problem: Nothing -> pure (Set.empty, ...
GlobalPackages (i.e: Packages that aren't ProjectPackages or DepPackages) are getting lumped into this group, such as process. Because of this, the deps for these packages are essentially truncated.

I don't know if this is intended behavior or not. If not, then I guess this is a glitch and we can just modify the code to include the dependencies for these packages. If it is intended, then we can make an option as @dbaynard said above if that seems ok?

[qrilka]

@vanceism7 the current version of that code is more or less my refactoring of the previous version to use new source maps. As far as I understand it you're correct, probably you've noticed that for wiredInPackages we do consult globalDumpMap to get package information. Now we have global packages in smGlobal field of a SourceMap so we could just do that for any of those packages.

[vanceism7]

Oh gotcha ok cool, I'll work on the PR for this then

Edit
Actually I think I did notice the smGlobal in the sourceMap. But the problem I was having was loading up dependencies for GlobalPackage. ProjectPackages and DepPackages have access to CommonPackage, but GlobalPackage doesn't hold any info except the version. I got a preliminary version of this working by treating global packages the same as wired in, I would load the deps from the globalDumpMap. Are there any ways to transform GlobalPackage into something else that can be worked with?

[qrilka]

@snoyberg I think something wasn't merged correctly here - compare https://svgur.com/s/Bq3 vs https://svgur.com/s/Bqb
The former version is from 1b1c2af and the latter is from the latest master 77aa9c3
I'll try to find out how did we get this

[snoyberg]

@qrilka what project did you generate those graphs from?

[qrilka]

@snoyberg I just run stack new testproject and added some boot packages into dependencies of package.yaml, namely:

dependencies:
- base >= 4.7 && < 5
- ghc
- xhtml
- haskeline
- Cabal
- mtl
- ghc-heap

then it's just stack dot + dot

[qrilka]

It's unclear how could that happen but I can't reproduce those incomplete graphs anymore.