Speed up CI

[fschutt]

Work in progress PR to get the CI runs to be more manageable

Goals:

• migrate from bash / python scripts to cargo xtask to be able to use Rust to script the CI for reproducability
• add more [jobs] to be able to run multiple CI steps in parallel
• Reduce CI times to 20 minutes or less, not 1h 30min

[syrusakbary]

As we chatted, let's keep xtask/makefile completely separated from the PR.
This PR should mainly focus on improving the speed of CI

[fschutt]

Results:

• parallelization: done
• sccache: -10 minutes, should be kept
• GitHub cache: obligatory speedup of ~20 minutes, should be kept
• wasmer fails to run on windows-latest, but not on windows-2019
• 8 cores: 2x speedup, but macos is the blocking part since we don't have 8-core runners, but we get faster failures
• cannot upgrade to macos-12, sadly
• bors: will only speed up merging, not being run in CI, removed for now since we can use GitHub auto-merge

Best Run: 35 minutes with 8 cores, sccache + GitHub cache

[syrusakbary]

Test C-API once build C-API is done should be close to instant (or very fast).

Right now is recompiling the c-api somehow? We should not do that

Also, the build c-api on windows should be part of the build matrix

[fschutt]

Okay so since the last signoff there were a couple of changes:

• Linking the CRT statically on Windows caused the tests to recompile, even if the feature-flags were the same.
• build-capi-windows-gnu is now part of the build matrix
• tried to cache the windows-x64 build, but failed
• fixed all caches, but R2 is often times locking up and taking 30mins to just retrieve the cache. I'd like to test later on with S3 instead of R2, but for now it's good enough
• upgraded to ubuntu-22.04 to avoid GitHub brownouts of ubuntu-18.04

[syrusakbary]

There's something clearly wrong with the cache here:

https://github.com/wasmerio/wasmer/actions/runs/3741028179/jobs/6350105848

[fschutt]

@syrusakbary Yes, I know, but I can't do anything about it, I just call the GH action with the proper parameters. If R2 decides to lock up, that's not really something I can debug. I can only try to use S3 to compare the reliability against R2.

[fschutt]

I'm currently blocked because I can't create an AWS account because I lost my 2FA access to my bank account. I can get it back, but only after the Dec 25th. Sorry :/

[syrusakbary]

Can we add integration tests on the steps defined here making sure that everything works as we laid out? (both for the wasm case and the webc case)

#3436 (comment)

Should be done.