How do compiler tests interact with editions?

[QuietMisdreavus]

During the Rust All-Hands last week, a question was brought up about how rustdoc would test its --edition flag to ensure that things that should compile in specific editions are tested to ensure this. Something like this should also be considered for the regular compiler tests as well.

A small plan i'm pondering, after a discussion with @nikomatsakis last week:

• Run tests in the newest edition by default
• Add an edition flag to compiletest so that certain tests can request a specific edition
• Add some functionality to compiletest to compile certain tests in multiple editions?
  • Running everything in every edition is a combinatoric explosion that would blow up our test times, but running specific tests in multiple editions is probably useful

[oli-obk]

Maybe we should run all test combinations once every beta release similar to the crater runs

[pnkfelix]

Related bug (I think) is #52979, which proposed extending our test suite to run every test (unless they opt out) under both the 2015 and 2018 editions.

• I was planning to leverage the compare-mode system that we used for NLL to implement this.
• I found in my own development that too many tests would cause failures; we would probably want to automatically run rustfix (and then compile the result) as part of the automatic test infrastructure.

I would recommend we still investigate something like this. We wouldn't need to test every edition, if for some reason we were worried about testing time growing too fast every three years; it would probably suffice to test just the latest two (i.e. the one under development and its predecessor).

[petrochenkov]

Please don't repeat the story with NLL testing (everything is run twice), it's awful and I'm looking forward to removing the NLL run at last.

Most of tests are edition-agnostic, and those few that may want to run on both editions can be annotated explicitly.
(We can make one initial "both editions for all tests" run of course.)

[petrochenkov]

If all tests are run for both editions, they at lest need to be organized into separate test suites (similarly to pretty tests), so you don't have to run both suites on e.g. x.py test src/test/ui.

src/test/ui may run all tests with their "default edition", and additional test suite like x.py test src/test/ui-edition or something runs tests with their "secondary edition" and is enabled only on CI.
The "default edition" for a test can be determined by some global default + test-local override.

[pnkfelix]

(I don't know how to check for unexpected fallout besides running everything twice. Of course its possible that we made a mistake in policy when we turned the compare-mode=nll ui suite variant on by default for everyone. Or maybe the mistake was just that we didn't provide an easy way to opt out of it as a local developer? We did check ahead of time with the infrastructure team about whether running the ui tests twice would impose a huge burden on the CI system...)

[jieyouxu]

compiletest triage: maybe there can be some kind of --force-all-editions flag for compiletest that will run every test for each edition, but I suspect that's going to produce a bunch of failures just from slightly different diagnostics / extra/fewer things here and there. I certainly do not want to be triaging a full CI run for such. I'll note that the --pass=check is already quite annoying and some times have to non-obviously write //@ ignore-pass specifically for that purpose. I don't think we should be adding more complications than that.