Add tests to rustbuild

[Mark-Simulacrum]

In order to run tests, we cfg out various parts of rustbuild. Generally
speaking, these are filesystem and process-spawning operations. Then, rustbuild
is run "as normal" and the various steps that where run are retrieved from the
cache and checked against the expected results.

Note that this means that the current implementation primarily tests "what" we
build, but doesn't actually test that what we build will build. In other
words, it doesn't do any form of dependency verification for any crate. This is
possible to implement, but is considered future work.

This implementation strives to cfg out as little code as possible; it also does
not currently test anywhere near all of rustbuild. The current tests are also
not checked for "correctness," rather, they simply represent what we do as of
this commit, which may be wrong.

Test cases are drawn from the old implementation of rustbuild, though the
expected results may vary.

r? @alexcrichton

[bors]

☔ The latest upstream changes (presumably #48599) made this pull request unmergeable. Please resolve the merge conflicts.

[Mark-Simulacrum]

Rebased and fixed tidy.

[bors]

☔ The latest upstream changes (presumably #48549) made this pull request unmergeable. Please resolve the merge conflicts.

[Mark-Simulacrum]

Rebased again.

[alexcrichton]

Taking a look at this now, nice!

I'm personally hesitant to land though due to the sprinkling of test-related cfg! macros and such, though. This seems like it'd be difficult to maintain over time and be a source of pain for changing rustbuild as well. (aka what if one of these files needs to be valid?).

Could we perhaps explore an alternate solution where something like run is stubbed out at a very low level? Ideally the testing here would be as least invasive as possible. Unfortunately though the design here is sort of inherently not-easily-testable...

[Mark-Simulacrum]

Stubbing out run isn't really practical as we call ensure from it (which is pretty much required for the testing design here, and I think any viable testing design). We could explore an alternate design that separated out ensure calls from the run method but I don't personally think that would work very well with how rustbuild is designed today.

I'm personally hesitant to land though due to the sprinkling of test-related cfg! macros and such, though. This seems like it'd be difficult to maintain over time and be a source of pain for changing rustbuild as well.

I agree with this in general but I think the overhead/pain is somewhat worth it as I feel the tests outweigh the pain caused by having the cfg!s. Ideally, we'd find some solution that wouldn't require them, but so far I haven't come up with anything that shows promise. Most of them are related directly to running executables or filesystem related operations, which both seem like something we'd basically need cfg!s for today.

Could we perhaps explore an alternate solution where something like run is stubbed out at a very low level? Ideally the testing here would be as least invasive as possible. Unfortunately though the design here is sort of inherently not-easily-testable...

I'm not entirely sure what you mean by stubbing out run, as that's mostly what this does anyway (the cfg! macros in Steps are in the steps that would be stubbed out, with basically the same relative diff).

One solution we might potentially see is pulling the ensure calls for each step out of run (and moving ensure off of builder perhaps so that we can guarantee that it isn't run from run) and then passing the results into the run method. I feel like this will get rather ugly fairly quickly due to code duplication and having to follow separate codepaths but I could see it maybe working out okay. Would you like to see me explore this route in perhaps a separate branch/PR? I don't think it'll work out well but we could see, I guess.

[alexcrichton]

Oh sorry I meant stubbing out run for processes instead of steps which should hopefully be lower down and the costliest part of rustbuild mostly. I'm not sure I'd agree though that these test outweight the cfg!, in the sense that we're almost no longer testing rustbuild at that point because the code is so heavily modified in test mode. This seems like it'd cause more effort to keep tests running than to diagnose/fix regressions?

I think it's a good idea to test rustbuild but I'd want to be sure that it'd done so in a relatively non-invasive way. Stubbing out running processes at the very core would be an example, but I'm also not sure how feasible that would be unfortunately :(

Other alternate designs (like the old compute-steps-first method) are more testable but have their own drawbacks too (hence the rewrite!). I'm not sure how easily testable the declarative design we have now is going to be :(

[Mark-Simulacrum]

Oh sorry I meant stubbing out run for processes instead of steps which should hopefully be lower down and the costliest part of rustbuild mostly.

This is true (and already done in this implementation). However, we also need/want to stub out filesystem access since we want to avoid creating files all over the build directory that are test-specific, and we ideally don't want to have to run tests single threaded or some other "solution" that avoids conflicts between tests.

I would like to note that the cfg! macros are actually fairly minimally sprinkled, I've annotated them below with why they are there. Reading through the list I think it's perhaps a little more than the ideal but actually looks fairly reasonable and maintainable -- there's very few edge cases I think.

src/bootstrap/compile.rs
692:        if cfg!(test) { // codegen backends -- we expect at least one to have been built, but we build none as we haven't run cargo
743:        if cfg!(test) { return; } // parsing filesystem path
1003:    if cfg!(test) { return Vec::new(); } // cargo is run via `Command::spawn` instead of traditional APIs

src/bootstrap/lib.rs
447:        if !cfg!(test) { // parsing command output, will panic if unexpected output
675:        if cfg!(test) { return; } // running command
682:        if cfg!(test) { return; } // running command
691:        if cfg!(test) { return true; } // running command
700:        if cfg!(test) { return true; } // running command
998:        if cfg!(test) { return String::from("0.1.2"); } // parsing file system

src/bootstrap/native.rs # these are all because we can't easily stub out the cc crate's run
62:        if cfg!(test) {
339:        if cfg!(test) {
395:        if cfg!(test) {
450:        if cfg!(test) {

src/bootstrap/util.rs
213:    if cfg!(test) { return Ok(()); } # the windows code uses low-level APIs that we can't really stub out

[bors]

☔ The latest upstream changes (presumably #48097) made this pull request unmergeable. Please resolve the merge conflicts.

[shepmaster]

Ping from triage, @alexcrichton — will you have time to continue your review soon?

[shepmaster]

@Mark-Simulacrum you have some conflicts as well.

[Mark-Simulacrum]

I probably won't rebase and hope to discuss this at the all-hands with @alexcrichton, so we might hold off until then.

[alexcrichton]

I'm down with that!

[Mark-Simulacrum]

I've reimplemented and pushed a new version that goes with the dry_run approach I've discussed with @alexcrichton, so I think this is ready for another review pass.

[alexcrichton]

Looks great! r=me with travis passing

[Mark-Simulacrum]

@bors r=alexcrichton

[bors]

📌 Commit 5556def has been approved by alexcrichton

[bors]

⌛ Testing commit 5556def81df8062a7553ad3a9668b71d2eb48445 with merge e640148acd9eacd3095d682dd8d2370ae564420f...

[bors]

💔 Test failed - status-appveyor

[Mark-Simulacrum]

@bors r=alexcrichton

[bors]

📌 Commit 184d3bc has been approved by alexcrichton

[kennytm]

@bors p=75

[bors]

⌛ Testing commit 184d3bc with merge 52af892...

[bors]

💔 Test failed - status-travis

[TimNN]

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

[TimNN]

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

[kennytm]

@bors retry

3 hour timeout in x86_64-gnu-incremental. The log has been corrupted by Travis.

[bors]

⌛ Testing commit 184d3bc with merge 17fea66...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 17fea66 to master...