List global package deps in stack dot graph

ChangeLog.md

@@ -16,6 +16,7 @@ Major changes:
     * Support for archives and repos in the `packages` section has
       been removed. Instead, you must use `extra-deps` for such
       dependencies. `packages` now only supports local filepaths.
+    * Add support for Git repositories containing (recursive) submodules.
     * Addition of new configuration options for specifying a "pantry
       tree" key, which provides more reproducibility around builds,
       and (in the future) will be used for more efficient package
@@ -56,6 +57,9 @@ Behavior changes:
   directory could affect interpretation of the script. See
   [#4538](https://github.com/commercialhaskell/stack/pull/4538)
 
+* When using `stack script`, custom snapshot files will be resolved
+  relative to the directory containing the script.
+
 Other enhancements:
 
 * Defer loading up of files for local packages. This allows us to get
@@ -80,6 +84,8 @@ Other enhancements:
 * Stack parses and respects the `preferred-versions` information from
   Hackage for choosing latest version of a package in some cases,
   e.g. `stack unpack packagename`.
+* The components output in the `The main module to load is ambiguous` message
+  now include package names so they can be more easily copy-pasted.
 * Git repos are shared across multiple projects. See
   [#3551](https://github.com/commercialhaskell/stack/issues/3551)
 * Use en_US.UTF-8 locale by default in pure Nix mode so programs won't
@@ -98,6 +104,20 @@ Other enhancements:
   [#4535](https://github.com/commercialhaskell/stack/issues/4535)/
 * Show snapshot being used when `stack ghci` is invoked outside of a project directory. See
   [#3651](https://github.com/commercialhaskell/stack/issues/3651)
+* The script interpreter now accepts a `--extra-dep` flag for adding
+  packages not present in the snapshot. Currently, this only works
+  with packages from Hackage, not Git repos or archives.
+* When using the script interpreter with `--optimize` or `--compile`,
+  Stack will perform an optimization of checking whether a newer
+  executable exists, making reruns significantly faster. There's a
+  downside to this, however: if you have a multifile script, and
+  change one of the dependency modules, Stack will not automatically
+  detect and recompile.
+* `stack clean` will delete the entire `.stack-work/dist` directory,
+  not just the relevant subdirectory for the current GHC version. See
+  [#4480](https://github.com/commercialhaskell/stack/issues/4480).
+* Add `stack purge` as a shortcut for `stack clean --full`. See
+  [#3863](https://github.com/commercialhaskell/stack/issues/3863).
 
 Bug fixes:
 

doc/GUIDE.md

@@ -280,6 +280,24 @@ As you can see from that path (and as emphasized earlier), the installation is
 placed to not interfere with any other GHC installation, whether system-wide or
 even different GHC versions installed by stack.
 
+## Cleaning your project
+
+You can clean up build artifacts for your project using the `stack clean` and `stack purge` commands.
+
+### `stack clean`
+
+`stack clean` deletes the local working directories containing compiler output.
+By default, that means the contents of directories in `.stack-work/dist`, for all the `.stack-work` directories within a project.
+
+Use `stack clean <specific-package>` to delete the output for the package _specific-package_ only.
+
+### `stack purge`
+
+`stack purge` deletes the local stack working directories, including extra-deps, git dependencies and the compiler output (including logs).
+It does not delete any snapshot packages, compilers or programs installed using `stack install`. This essentially
+reverts the project to a completely fresh state, as if it had never been built.
+`stack purge` is just a shortcut for `stack clean --full`
+
 ### The build command
 
 The build command is the heart and soul of stack. It is the engine that powers

doc/install_and_upgrade.md

@@ -21,7 +21,7 @@ future, we are open to supporting more OSes (to request one, please
 Binary packages are signed with this [signing key](SIGNING_KEY.md).
 
 If you are writing a script that needs to download the latest binary, you can
-use URLs like `https://get.haskellstack.org/stable/<PLATFORM>.<EXTENSION>` (e.g. //get.haskellstack.org/stable/linux-x86_64.tar.gz) that always point to the latest stable release.
+use URLs like `https://get.haskellstack.org/stable/<PLATFORM>.<EXTENSION>` (e.g. https://get.haskellstack.org/stable/linux-x86_64.tar.gz) that always point to the latest stable release.
 
 ## Windows
 

doc/lock_files.md

@@ -0,0 +1,241 @@
+<div class="hidden-warning"><a href="https://docs.haskellstack.org/"><img src="https://cdn.jsdelivr.net/gh/commercialhaskell/stack/doc/img/hidden-warning.svg"></a></div>
+
+# Lock Files
+
+Stack attempts to provide reproducible build plans. This involves
+reproducibly getting the exact same contents of source packages and
+configuration options (like cabal flags and GHC options) for a given
+set of input files. There are a few problems with making this work:
+
+* Entering all of the information to fully provide reproducibility is
+  tedious. This would include things like Hackage revisions, hashes of
+  remote tarballs, etc. Users don't want to enter this information.
+* Many operations in Stack rely upon a "snapshot hash," which
+  transitively includes the completed information for all of these
+  dependencies. If any of that information is missing when parsing the
+  `stack.yaml` file or snapshot files, it could be expensive for Stack
+  to calculate it.
+
+To address this, we follow the (fairly standard) approach of having a
+_lock file_. The goal of the lock file is to cache completed
+information about all packages and snapshot files so that:
+
+* These files can be stored in source control
+* Users on other machines can reuse these lock files and get identical
+  build plans
+* Rerunning `stack build` in the future is deterministic in the build
+  plan, not depending on mutable state in the world like Hackage
+  revisions
+    * **NOTE** If, for example, a tarball available remotely is
+      deleted or the hash changes, it will not be possible for Stack
+      to perform the build. However, by deterministic, we mean it
+      either performs the same build or fails, never accidentally
+      doing something different.
+* Stack can quickly determine the build plan in the common case of no
+  changes to `stack.yaml` or snapshot files
+
+This document explains the contents of a lock file, how they are used,
+and how they are created and updated.
+
+## stack.yaml and snapshot files
+
+Relevant to this discussion, the `stack.yaml` file specifies:
+
+* Resolver (the parent snapshot)
+* Compiler override
+* `extra-deps`
+* Flags
+* GHC options
+* Hidden packages
+
+The resolver can either specify a compiler version or another snapshot
+file. This snapshot file can contain the same information referenced
+above for a `stack.yaml`, with the following differences:
+
+* The `extra-deps` are called `packages`
+* Drop packages can be included
+
+Some of this information is, by its nature, complete. For example, the
+"flags" field cannot be influenced by anything outside of the file
+itself.
+
+On the other hand, some information in these files can be
+incomplete. Consider:
+
+```yaml
+resolver: lts-13.9
+packages: []
+extra-deps:
+- https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
+```
+
+This information is _incomplete_, since the contents of that URL may
+change in the future. Instead, you could specify enough information in
+the `stack.yaml` file to fully resolve that package. That looks like:
+
+```yaml
+extra-deps:
+- size: 1442
+  url: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
+  cabal-file:
+    size: 613
+    sha256: 2ba66a092a32593880a87fb00f3213762d7bca65a687d45965778deb8694c5d1
+  name: acme-missiles
+  version: '0.3'
+  sha256: e563d8b524017a06b32768c4db8eff1f822f3fb22a90320b7e414402647b735b
+  pantry-tree:
+    size: 226
+    sha256: 614bc0cca76937507ea0a5ccc17a504c997ce458d7f2f9e43b15a10c8eaeb033
+```
+
+Users don't particularly feel like writing all of that. Therefore,
+it's common to see _incomplete_ information in a `stack.yaml` file.
+
+Additionally, the `lts-13.9` information is _also_ incomplete. While
+we assume in general that LTS snapshots never change, there's nothing
+that technically prohibits that from happening. Instead, the complete
+version of that field is:
+
+```yaml
+resolver:
+  size: 496662
+  url: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/13/9.yaml
+  sha256: 83de9017d911cf7795f19353dba4d04bd24cd40622b7567ff61fc3f7223aa3ea
+```
+
+Also something people don't feel like writing by hand.
+
+## Recursive snapshot layers
+
+Snapshot files can be _recursive_, where `stack.yaml` refers to
+`foo.yaml`, which refers to `bar.yaml`, which refers to `baz.yaml`. A
+local snapshot file can refer to a remote snapshot file (available via
+an HTTP(S) URL).
+
+We need to encode information from _all_ of these snapshot layers and
+the `stack.yaml` file in the lock file, to ensure that we can detect
+if anything changes.
+
+## Performance
+
+In addition to acting as a pure correctness mechanism, the design of a
+lock file given here also works as a performance improvement. Instead
+of requiring that all snapshot files be fully parsed on each Stack
+invocation, we can store information in the lock file and bypass
+parsing of the additional files in the common case of no changes.
+
+## Lock file contents
+
+The lock file contains the following information:
+
+* The full snapshot definition information, including completed
+  package locations for both `extra-deps` and packages in
+  snapshot files
+    * **NOTE** This only applies to _immutable_ packages. Mutable
+      packages are not included in the lock file.
+* Completed information for the snapshot locations
+* A hash of the `stack.yaml` file
+* The snapshot hash, to bypass the need to recalculate this on each
+  run of Stack
+
+It looks like the following:
+
+```yaml
+# Lock file, some message about the file being auto-generated
+stack-yaml:
+  sha256: XXXX
+  size: XXXX # in bytes
+
+snapshots:
+  # Starts with the snapshot specified in stack.yaml,
+  # then continues with the snapshot specified in each
+  # subsequent snapshot file
+  - original:
+      foo.yaml # raw content specified in a snapshot file
+    completed:
+      file: foo.yaml
+      sha256: XXXX
+      size: XXXX
+  - original:
+      lts-13.9
+    completed:
+      size: 496662
+      url: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/13/9.yaml
+      sha256: 83de9017d911cf7795f19353dba4d04bd24cd40622b7567ff61fc3f7223aa3ea
+
+compiler: ghc-X.Y.Z
+
+packages:
+  acme-missiles:
+    location:
+      # QUESTION: any reason we need to specify which snapshot file it came from? I don't think so...
+      original: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
+      completed:
+        size: 1442
+        url: https://hackage.haskell.org/package/acme-missiles-0.3.tar.gz
+        cabal-file:
+          size: 613
+          sha256: 2ba66a092a32593880a87fb00f3213762d7bca65a687d45965778deb8694c5d1
+        name: acme-missiles
+        version: '0.3'
+        sha256: e563d8b524017a06b32768c4db8eff1f822f3fb22a90320b7e414402647b735b
+        pantry-tree:
+          size: 226
+          sha256: 614bc0cca76937507ea0a5ccc17a504c997ce458d7f2f9e43b15a10c8eaeb033
+    flags: ...
+    hidden: true/false
+    ghc-options: [...]
+```
+
+**NOTE** The `original` fields may seem superfluous at first. See the
+update procedure below for an explanation.
+
+## Creation
+
+Whenever a `stack.yaml` file is loaded, Stack checks for a lock file
+in the same file path, with a `.lock` extension added. For example, if
+you run `stack build --stack-yaml stack-11.yaml`, it will use a lock
+file in the location `stack-11.yaml.lock`. For the rest of this
+document, we'll assume that the files are simply `stack.yaml` and
+`stack.yaml.lock`.
+
+If the lock file does not exist, it will be created by:
+
+* Loading the `stack.yaml`
+* Loading all snapshot files
+* Completing all missing information
+* Writing out the new `stack.yaml.lock` file
+
+## Dirtiness checking
+
+If the `stack.yaml.lock` file exists, its last modification time is
+compared against the last modification time of the `stack.yaml` file
+and any local snapshot files. If any of those files is more recent
+than the `stack.yaml` file, and the file hashes in the lock file
+do not match the files on the filesystem, then the update procedure is
+triggered. Otherwise, the `stack.yaml.lock` file can be used as the
+definition of the snapshot.
+
+## Update procedure
+
+The simplest possible implementation is: ignore the lock file entirely
+and create a new one followign the creation steps above. There's a
+significant downside to this, however: it may cause a larger delta in
+the lock file than intended, by causing more packages to be
+updates. For example, many packages from Hackage may have their
+Hackage revision information updated unnecessarily.
+
+The more complicated update procedure is described below. **QUESTION**
+Do we want to go the easy way at first and later implement the more
+complicated update procedure?
+
+1. Create a map from original package location to completed package
+   location in the lock file
+2. Load up each snapshot file
+3. For each incomplete package location:
+    * Lookup the value in the map created in (1)
+    * If present: use that completed information
+    * Otherwise: complete the information using the same completion
+      procedure from Pantry as in "creation"
+
+This should minimize the number of changes to packages incurred.

doc/pantry.md

@@ -306,8 +306,6 @@ directories is available in snapshots to ensure reproducibility.
 resolver: lts-8.21 # Inherits GHC version and package set
 compiler: ghc-8.0.1 # Overwrites GHC version in the resolver, optional
 
-name: my-snapshot # User-friendly name
-
 # Additional packages, follows extra-deps syntax
 packages:
 - unordered-containers-0.2.7.1
@@ -371,7 +369,6 @@ packages:
   pantry-tree:
     size: 7376
     sha256: ac2601c49cf7bc0f5d66b2793eddc8352f51a6ee989980827a0d0d8169700a03
-name: my-snapshot
 hidden:
   warp: false
   wai: true

package.yaml

@@ -242,7 +242,6 @@ library:
   - Stack.Types.NamedComponent
   - Stack.Types.Nix
   - Stack.Types.Package
-  - Stack.Types.PackageDump
   - Stack.Types.PackageName
   - Stack.Types.Resolver
   - Stack.Types.Runner

snapshot.yaml

@@ -18,6 +18,7 @@ packages:
 - unliftio-0.2.8.0@sha256:5a47f12ffcee837215c67b05abf35dffb792096564a6f81652d75a54668224cd,2250
 - happy-1.19.9@sha256:f8c774230735a390c287b2980cfcd2703d24d8dde85a01ea721b7b4b4c82944f,4667
 - fsnotify-0.3.0.1@rev:1
+- process-1.6.3.0@sha256:fc77cfe75a9653b8c54ae455ead8c06cb8adc4d7a340984d84d8ca880b579919,2370 #because of https://github.com/haskell/process/pull/101
 
 flags:
   cabal-install:

src/Options/Applicative/Complicated.hs

@@ -79,11 +79,12 @@ addCommand :: String   -- ^ command string
            -> String   -- ^ title of command
            -> String   -- ^ footer of command help
            -> (a -> b) -- ^ constructor to wrap up command in common data type
+           -> (a -> c -> c) -- ^ extend common settings from local settings
            -> Parser c -- ^ common parser
            -> Parser a -- ^ command parser
            -> ExceptT b (Writer (Mod CommandFields (b,c))) ()
-addCommand cmd title footerStr constr =
-  addCommand' cmd title footerStr (\a c -> (constr a,c))
+addCommand cmd title footerStr constr extendCommon =
+  addCommand' cmd title footerStr (\a c -> (constr a,extendCommon a c))
 
 -- | Add a command that takes sub-commands to the options dispatcher.
 addSubCommands