Support using stackage snapshots with GHCJS

[mgsloan]

Currently, stack only supports GHCJS via a compiler resolver that looks like ghcjs-0.1.0.20150924_ghc-7.10.2.

I can think of a couple ways to add support:

1. Add a --ghcjs flag and ghcjs: True config field, for use with snapshot resolvers. This is consistent with needing to pass in --ghcjs to cabal.

2. Have some sort of snapshot resolver modifier which specifies to use GHCJS. For example ghcjs-lts-3.9. Note that there wouldn't be an actual snapshot, though. This would preserve the property that the resolver specifies which compiler is being used. Not sure if this is a big deal or not, but having compiler revolvers suggests that the resolver specifies a particular compiler.

3. Add a ghcjs-packages field which lists the packages to build with GHCJS. This would be a more involved change. This would simplify building client and server in one go and ensure that your serverside code and clientside code are using the same dependencies - no need to make sure your resolvers match. Some notes on this:

• The change would be pretty substantial. Currently, we know what compiler we're using when executing a stack command in the presence of a build config. This would make it more of a thing that's part of planning / executing the build.
• It would need to be decided whether packages compiled by both GHC and GHCJS should need to be listed in ghcjs-packages. We could figure out all the local packages depended on by ghcjs packages, but I imagine some code would need to be written for that.
• solver would use --ghcjs with cabal whenever there's a ghcjs package. This would mean that we'd be using GHCJS's global DB, which could potentially have different versions (but hopefully not)
• ghci would need to take a --ghcjs flag for cases where the targets can either be built with GHC or GHCJS.

It's also worthwhile to consider what to do in similar situations in the future, where we have ghc based compilers which should be used to build some of your local packages. Perhaps if we put the effort into doing (3), it should be generalized?

I'm leaning towards (1) for consistency with the existing GHCJS ecosystem / philosophy. This way, if an application works with ghcjs as well as ghc, you can just do a stack build --ghcjs. I don't think the effort and complexity of (3) is worth it. It does mean that the compiler is no longer determined by the resolver. But, in a sense, this was always the case since we don't demand a particular GHC version (unless compiler-check: match-exact).

[3noch]

I'm highly ignorant of the GHCJS ecosystem, so I might be way off base: Wouldn't using the exact same snapshot (except for compiler) for both GHC and GHCJS projects be problematic? Many packages that target GHCJS will optionally depend on GTK+ when built with GHC. So your GHCJS snapshots would need to include GTK+ bindings, which won't build. Still, I can certainly see the value of having GHCJS snapshots mirror their GHC counterparts. I would be slightly concerned that tests that pass with one compiler might fail for another (especially considering the tests may differ between the two).

It seems to me that you would still be better off separating GHC and GHCJS snapshots from each other, but with the added caveat that they should mirror each other for the packages that they do have in common.

Also, considering that no GHCJS code and GHC code will ever need to "link," there is really no reason to fear having two entirely different snapshot versions for each of them (e.g. a server package using LTS-3.9 and a front-end package using GHCJS-LTS-3.8). No doubt, this might not be ideal purely from "consistency" perspective, but I don't see any technical dangers arising from it.

With all this in mind, if you wanted stack to support having the same stack.yaml file define how to build an entire system (with some packages using GHC and others using GHCJS), it seems that the the resolver configuration then needs to live beneath another, even broader setting, like "ecosystem" or something. So a stack.yaml file might look like this:

ecosystem: ghc
  - resolver: LTS-3.9
ecosystem: ghcjs
  - resolver: GHCJS-LTS-3.8

(Of course, the ghc ecosystem could be assumed in most cases.)

This is likely a terrible idea (because I'm way out of the loop on this stuff), but you get the idea.

P.S. Great work. I'm so excited where all this is going. I recently compiled reflex-todomvc with stack. Amazing.

[mgsloan]

I'm highly ignorant of the GHCJS ecosystem, so I might be way off base: Wouldn't using the exact same snapshot (except for compiler) for both GHC and GHCJS projects be problematic? Many packages that target GHCJS will optionally depend on GTK+ when built with GHC. So your GHCJS snapshots would need to include GTK+ bindings, which won't build.

I thought this as well back when adding the initial support in stack. However, it turns out that GHCJS will happily build most of stackage, because C FFI calls get turned into javascript calls. This means that you can sometimes add support for a library that depends on FFI by just adding some javascript. The main circumstance I can think of where GHCJS won't be able to compile some code is if TH runs some function that uses C FFI, but doesn't have a JS implementation. This should be rare indeed! It seems rather messy when TH uses IO..

So, I would expect GHCJS to be able to build GTK+, yielding something riddled with runtime errors, though I haven't tested that. Note that if you're using GHCJS, then ghcjs-dom will target JS and not depend on GTK+: https://github.com/ghcjs/ghcjs-dom/blob/master/ghcjs-dom.cabal#L565

One thing that convinced me that this is all quite reasonable is that stackage already has packages which only work on a particular operating system. You could almost think of GHCJS as providing a different OS environment.

We'll want to have all GHCJS-only packages include something like if !impl(ghcjs):, buildable: False in their cabal files. This way, we'll know to not expect them to build with GHC.

I would be slightly concerned that tests that pass with one compiler might fail for another (especially considering the tests may differ between the two).

Yes, we'll likely need a way to add additional expected-failures for GHCJS, in build-constraints.yaml. This list of expected failures might be an interesting source of information for making GHCJS behavior even closer to GHC's behavior.

Also, considering that no GHCJS code and GHC code will ever need to "link," there is really no reason to fear having two entirely different snapshot versions for each of them (e.g. a server package using LTS-3.9 and a front-end package using GHCJS-LTS-3.8). No doubt, this might not be ideal purely from "consistency" perspective, but I don't see any technical dangers arising from it.

There are certainly technical dangers. For example, if you depend on two different versions of aeson, the serialization of datatypes shared between your client and server can differ. There's also the technical convenience of not needing to extend all the stackage infrastructure to handle GHCJS-specific snapshots.

P.S. Great work. I'm so excited where all this is going. I recently compiled reflex-todomvc with stack. Amazing.

Glad it's working well for you! GHCJS is certainly amazing stuff. I hope first class support in stack will make it easy to get started with it.

[mgsloan]

I think it will make sense to be able to override the version of GHCJS that's in the snapshot. Until GHCJS is in stackage, this will also allow GHCJS to be used with a snapshot.

How about ghcjs-version: 0.2.0? It'll use the GHC version of the current resolver to determine the full GHCJS version. I don't think it would make sense to use extra-deps for this.

[3noch]

You've addressed my concerns very well.

[mgsloan]

I've discussed this with @snoyberg, and we're going to go with adding a compiler: configuration field, which uses the same format as compiler resolvers. This is quite useful for things beyond GHCJS, but for the GHCJS case it means you'll be able to do this:

resolver: lts-3.10
compiler: ghcjs-0.1.0.20150923_ghc-7.10.2

Once GHCJS is released to hackage and made part of stackage, we can consider adding a more convenient way to specify that we want to use the GHCJS from the snapshot.

[mgsloan]

Ok, this now works!!

To use, make your stack.yaml look something like this:

resolver: lts-3.10
compiler: ghcjs-0.1.0.20150924_ghc-7.10.2
compiler-check: match-exact

More docs / tutorial coming later.