SDist code deling with custom-setup incorrect

[istathar]

When uploading http-streams using stack upload . (ie the sdist step) I get the following warning:

Warning: In getPackageArgs: custom-setup in use, but no dependency map present
Could not find custom-setup dep: Cabal
Could not find custom-setup dep: base

which is weird because the .cabal file in question has what I was told to put there previously:

...
custom-setup
  setup-depends:     base >= 4.5,
                     Cabal >= 1.24

...

I asked in the Haskell channel at work and the epic @SamRoberts read through the stack code and in short order found the message is emitted from Execute.hs's function withSingleContext. Its fourth argument is a Maybe (Map something something).

This appears to be called from SDist.hs where the fourth argument is hard coded to Nothing. Hence warning.

Seems either the Nothing is wrong and we need to pass in the custom-setup (?) map or the warning code needs to know that it's not going to actually get a map and not warn.

AfC

feel free to change the issue title to something more helpful

[xaviershay]

I got this warning too and don't know what to do about it.

[dbaynard]

@afcowie @SamRoberts rather than the Nothing in line 331, take a look at line 329

stack/src/Stack/SDist.hs

Lines 328 to 331 in 9900d99

	withExecuteEnv bopts boptsCli baseConfigOpts locals
	[] [] [] -- provide empty list of globals. This is a hack around custom Setup.hs files
	$ \ee ->
	withSingleContext ac ee task Nothing (Just "sdist") $ \_package cabalfp _pkgDir cabal _announce _outputType -> do

The change was introduced to fix #941. I'll investigate.

Edit: The question is, why isn't this branch being taken, instead of the branch which logs that error.

stack/src/Stack/Build/Execute.hs

Lines 1095 to 1122 in 9900d99

	-- This branch is usually taken for builds, and
	-- is always taken for `stack sdist`.
	--
	-- This approach is debatable. It adds access to the
	-- snapshot package database for Cabal. There are two
	-- possible objections:
	--
	-- 1. This doesn't isolate the build enough; arbitrary
	-- other packages available could cause the build to
	-- succeed or fail.
	--
	-- 2. This doesn't provide enough packages: we should also
	-- include the local database when building local packages.
	--
	-- Currently, this branch is only taken via `stack
	-- sdist` or when explicitly requested in the
	-- stack.yaml file.
	(Nothing, _) -> do
	warnCustomNoDeps
	return $ cabalPackageArg ++
	-- NOTE: This is different from
	-- packageDBArgs above in that it does not
	-- include the local database and does not
	-- pass in the -hide-all-packages argument
	("-clear-package-db"
	: "-global-package-db"
	: map (("-package-db=" ++) . toFilePathNoTrailingSep) (bcoExtraDBs eeBaseConfigOpts)
	++ ["-package-db=" ++ toFilePathNoTrailingSep (bcoSnapDB eeBaseConfigOpts)])

---

Meanwhile, more detail is generally helpful. Would you please indicate which stack version, paste the --verbose output, and link to the code causing the issue?

[dbaynard]

@afcowie if you remove the explicit setup-depends then the sdist step should pick up the correct branch?

[istathar]

@dbaynard I'll try it, but I'm pretty sure I put that there because stack told me to (a sdist warning) in the first place.

[dbaynard]

Quite — the bundled package will be missing the required build dependencies. But at least we can test the correct code path.

[maksbotan]

This is still relevant with 2.5.1, stack sdist can't pack packages with dependencies in custom-setup. Is there a hope for a proper fix?

[qrilka]

@maksbotan could you give me some example package using custom-setup which I could use to test this?

[qrilka]

http-streams doesn't appear to use it anymore

[maksbotan]

Actually, now I'm only able to reproduce this if I run stack sdist before stack build and if I don't have anything in cache for the snapshot I'm using. For example, I set lts-15.14, run stack build, run stack sdist and it succeeds. Then change it to lts-15.15, run only stack sdist and it fails.

Setup.hs:

import Data.Aeson
import Distribution.Simple
main = defaultMain

stack-example.cabal:

cabal-version: 2.0

name:           stack-example
version:        0.1.0.0
author:         Author name here
maintainer:     example@example.com
copyright:      2020 Author name here
license:        BSD3
build-type:     Custom

custom-setup
  setup-depends:
      Cabal
    , aeson
    , base
    , bytestring
    , directory
    , filepath
    , process

library
  exposed-modules: Lib
  hs-source-dirs: .
  build-depends: base >=4.7 && <5
  default-language: Haskell2010

You can use Lib.hs with any content, I have it from stack new.

I'm not sure why it didn't work (consistently) in our setup, as I certainly did not switch resolvers, but I workarounded it by using cabal new-sdist instead of stack.

[qrilka]

Thank you, this helps a lot. I will take a look. As for problems with consistency - probably there's some other unaccounted detail, we'll see

[maksbotan]

My guess is that subsequent run is able to get aeson from some package db somewhere 🤷‍♂️

Here's the command that stack builds Setup.hs with:

2020-12-07 19:10:33.221488: [debug] Run process within /Users/maksbotan/temp/stack-example/: /Users/maksbotan/.stack/programs/x86_64-osx/ghc-8.8.3/bin/ghc-8.8.3 --make -odir /Users/maksbotan/temp/stack-example/.stack-work/dist/x86_64-osx/Cabal-3.0.1.0/setup -hidir /Users/maksbotan/temp/stack-example/.stack-work/dist/x86_64-osx/Cabal-3.0.1.0/setup -i -i. -clear-package-db -global-package-db -package-db=/Users/maksbotan/.stack/snapshots/x86_64-osx/10712ee20f8fbb479b4f6a44b3fdbf74efb78ff1266e32c1cdfc539b18e48fab/8.8.3/pkgdb -package-db=/Users/maksbotan/temp/stack-example/.stack-work/install/x86_64-osx/10712ee20f8fbb479b4f6a44b3fdbf74efb78ff1266e32c1cdfc539b18e48fab/8.8.3/pkgdb -hide-all-packages -package=Cabal -package=aeson -package=base -package=bytestring -package=directory -package=filepath -package=process -optP-include -optP/Users/maksbotan/temp/stack-example/.stack-work/dist/x86_64-osx/Cabal-3.0.1.0/setup/setup_macros.h /Users/maksbotan/temp/stack-example/Setup.hs /Users/maksbotan/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs -main-is StackSetupShim.mainOverride -o /Users/maksbotan/temp/stack-example/.stack-work/dist/x86_64-osx/Cabal-3.0.1.0/setup/setup -threaded

Note the -hide-all-packages -package=Cabal -package=aeson -package=base -package=bytestring -package=directory -package=filepath -package=process part.

[qrilka]

@maksbotan something's quite odd here - changing 15.14 to 15.15 brought me only error of cabal file with no description and only when I switched to 15.16 I see

2020-12-08 10:14:46.339449: [error] 
--  While building package sdist-custom-0.1.0.0 using:
      /home/kirill/.stack/programs/x86_64-linux/ghc-tinfo6-8.8.3/bin/ghc-8.8.3 --make -odir /home/kirill/ws/h/stack-tests/sdist-custom/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0/setup -hidir /home/kirill/ws/h/stack-tests/sdist-custom/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0/setup -i -i. -clear-package-db -global-package-db -package-db=/home/kirill/.stack/snapshots/x86_64-linux-tinfo6/518f90d8aa8fc89d62a36a2d698ab0d095d00cf4ee1e9cbfd906c17aea204f6c/8.8.3/pkgdb -package-db=/home/kirill/ws/h/stack-tests/sdist-custom/.stack-work/install/x86_64-linux-tinfo6/518f90d8aa8fc89d62a36a2d698ab0d095d00cf4ee1e9cbfd906c17aea204f6c/8.8.3/pkgdb -hide-all-packages -package=Cabal -package=aeson -package=base -package=bytestring -package=directory -package=filepath -package=process -optP-include -optP/home/kirill/ws/h/stack-tests/sdist-custom/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0/setup/setup_macros.h /home/kirill/ws/h/stack-tests/sdist-custom/Setup.hs /home/kirill/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs -main-is StackSetupShim.mainOverride -o /home/kirill/ws/h/stack-tests/sdist-custom/.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.0.1.0/setup/setup -threaded
    Process exited with code: ExitFailure 1

[qrilka]

It looks like there's nothing in stack sdist ensuring proper deps being present when they are required so the obvious workaround a the moment is to do an extra stack build --only-dependencies. I'll think about a proper solution for this issue.

[maksbotan]

For reference, I'm on macOS Catalina with

$ stack --version
Version 2.5.1, Git revision d6ab861544918185236cf826cb2028abb266d6d5 x86_64 hpack-0.33.0

However, our CI runs Ubuntu with stack 2.1 and experienced this as well.

[qrilka]

It doesn't look like OS could matter in any way and the code causing this was there at least in Stack-1.9.3.1 already, an implementation detail, so to say...