Build misc_testsuite from source

[marmistrz]

Now we require the compiler toolchain to have both the host target and the wasm32-wasi target. Issues with this PR:

• cargo clean doesn't remove misc_testsuite/target
• tests are being build during cargo build (cf. Add support for test build scripts rust-lang/cargo#1581)

Closes CraneStation/wasi-misc-tests#18

[kubkon]

I reckon that even with the issues listed in the PR's description, this is a much better setup than the previous one, so +1 from me.

cc @sunfishcode @alexcrichton

[sunfishcode]

Please also remove the exclude = ["misc_testsuite/*.wasm"] from the top-level Cargo.toml :-).

[sunfishcode]

Would it work to add a --target-dir to put this build's target somewhere inside OUT _DIR, or the outer cargo build's target dir, to fix cargo clean?

[marmistrz]

Done. I thought it would conflict with the cargo locks on build directory, but it appears it doesn't.

[kubkon]

@sunfishcode are you happy with the PR so that we could merge it? :-)

[marmistrz]

I thought about it for a while more, and there's one more problem with the current approach. If a test is broken, it may occur that it's impossible to make it pass in wasi-common, forcing us to do a revert. Adding a CI check here is a chicken&egg problem, so I suggest every PR to this repository should have a corresponding PR to wasi-common, bumping the revision of the submodule. (this can be a part of the PR template on GitHub)

Alternatively we can keep the tests in this repo, but it will make it harder to reuse the testsuite for other projects. (actually, I think I prefer this solution as a simpler one)

[kubkon]

Hmm, I’m not sure I see the problem here, but maybe if I explain how I understand it, you’ll be able to point me in the right direction. So i thought that, since wasi-misc-tests is referenced as a submodule, we’d always point at some stable rev and not necessarily the HEAD which minimises the probability of a broken test creeping in. Additionally, I’d also imagine that there’d be far more activity in this repo rather than wasi-misc-tests in terms of development. Currently, we do indeed go back and forth between the two a lot but I’d imagine it dying out somewhat as soon as we get the first full experimental support for all hostcalls on Windows. :-)

[marmistrz]

1. Test PR 1 lands. The authors forgets to check wasi-common and makes no wasi-common PR
2. Test PR 2 lands. The authors updates the submodule. Kaboom, the test from PR 1 fails.
3. We need to revert test PR 1, so that test PR 2 can be used in wasi-common at all.

It's true that there'll be much less activity here, it's why enforcing a matching PR to wasi-common during review is also ok.

[kubkon]

Ah, OK, I get it now. Thanks for the explanation :-) I feel that there is option 3 that doesn’t require creating a matching PR to this repo immediately after a new test case lands in wasi-misc-tests, nor having the two repos merged into one. Namely, if a test fails for whatever reason, and we don’t want to hinder development, we could always ignore it in build.rs :-)

[sunfishcode]

Yes, this looks good to me!