Stack test runs old test suites when build breaks rather than bailing

[rrnewton]

This happens with recent HEAD versions and with 0.1.2.

Steps to reproduce:

Take any successful building/testing project, then:

$ stack build
$ stack test
$ <put a type error in source file>
$ stack test

Expected behavior:

stack test would cause a rebuild, rebuild would fail with type error message, and that error would be the last thing left on the terminal, because testing would not continue.

Actual behavior:

Type error does appear on the terminal, but you have to scroll back to find it and know that it's there. Rather, stack test continues and executes the old (stale) test suite, giving a false sense of confidence by filling the screen with passing test output.

This is related to tickets concerning failure to rebuild (#481, #671, #691), but only slightly.

[snoyberg]

I'm unable to reproduce this, using the stack new template. Are you sure this isn't just a dupe of #481?

[rrnewton]

Ok. Let me dig a bit more to figure out when/where the above behavior happens for me. Good to know it wasn't the intended behavior tho.

[rrnewton]

Would 481 ever lead to errorful build output followed by stale test output on the terminal (resulting from a single stack test command)?

[snoyberg]

I can't really tell without seeing your output, but: if you know for certain that the same invocation produced both error output which prevented the test suite from compiling, and then ran the test suite, that's a new bug. But we'd need a repro to look into that, as I've never encountered that behavior despite attempts to trigger it.

[rrnewton]

Ok, the most I can reproduce right now is a failure to rebuild which I think is fixed. Specifically, the below commands:

git clone git@github.com:iu-parfunc/lvars.git --recursive
cd lvars
git checkout 6b9963660bec03de9fa635a3bbe560d32f35407d

stack --version
stack test

cd haskell/par-mergesort
stack test .
# Should see output, looks good.

echo blahblah >> Control/Par/MergeSort.hs

stack-0.1.2.0-x86_64-linux test .
# Prints good output again.  Fails to rebuild.

They show the test failing to do a rebuild which would have failed. But not actually doing the build and ignoring the bad exit status. (Which I really thought was happening, but I confess I may have misremembered.) Anyway, the above problem with 0.1.2 is fixed in HEAD so I'll close this for now.