Add ‘#[serial]’ tests (v2) #42684

shivjm · 2017-06-15T19:52:44Z

(Fixed version of #42626; sorry for the noise.)

This PR adds a #[serial] attribute that marks tests that should be executed one-by-one rather than concurrently (per #33519).

I’ve confirmed that the libtest and tidy tests pass. I wasn’t able to run the full suite as a lot of the run_make tests seem to fail on my Windows system, but everything passed up to that point.

Rust (and PR!) newbie here, so please do let me know if I’m doing something poorly or incorrectly!

rust-highfive · 2017-06-15T19:52:57Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

leonardo-m · 2017-06-15T20:26:58Z

Perhaps a name like #[serial_test] is better?

Mark-Simulacrum · 2017-06-15T20:32:30Z

I think #[serial] is fine given that #[test] will also be there, but I could see an argument being made that serial is too generic a name. I do think this needs a feature gate though, which I believe hasn't yet been implemented. I could be wrong though.

shivjm · 2017-06-15T21:51:18Z

Ah, I didn’t understand how the feature gates work and the example I followed (#42219) is apparently a WIP. Looking at #42088, I need to gate this by:

Adding an entry to the declare_features block in feature_gate.rs with a feature name, a version, and an issue number:

(active, serial_tests, "1.19.0", Some(42684))

Specifying AttributeGate::Gated for the serial attribute in BUILTIN_ATTRIBUTES (same file), which needs a feature name, an explanation, and a function to check its configuration (auto-generated using cfg_fn):

("serial", Normal, Gated(Stability::Unstable, "serial_tests", "the `#[serial]` attribute is an experimental feature", cfg_fn!(serial_tests))

Adding a new document under unstable-book/src/language-features/ that points to this PR.
Adding an entry under unstable-book/src/SUMMARY.md that points to that document.

Is this correct? Have I missed anything? I’ll try to get it done pronto, but if not, it will be about a week.

Re: the name of the attribute itself, I can change that if serial is too generic.

bjorn3

Few minor things. Overal code looks good to me.

bjorn3 · 2017-06-20T12:10:22Z

src/libsyntax/test.rs

@@ -52,7 +52,8 @@ struct Test {
    path: Vec<Ident> ,
    bench: bool,
    ignore: bool,
-    should_panic: ShouldPanic
+    should_panic: ShouldPanic,
+    serial: bool


Please end the line with a comma, so there is less noise when new fields get added.

bjorn3 · 2017-06-20T12:10:35Z

src/libsyntax/test.rs

@@ -133,7 +134,8 @@ impl<'a> fold::Folder for TestHarnessGenerator<'a> {
                        path: self.cx.path.clone(),
                        bench: is_bench_fn(&self.cx, &i),
                        ignore: is_ignored(&i),
-                        should_panic: should_panic(&i, &self.cx)
+                        should_panic: should_panic(&i, &self.cx),
+                        serial: is_serial(&i)


bjorn3 · 2017-06-20T12:11:51Z

src/libtest/lib.rs

+                    name: StaticTestName("first"),
+                    ignore: false,
+                    should_panic: ShouldPanic::No,
+                    serial: true


bjorn3 · 2017-06-20T12:12:06Z

src/libtest/lib.rs

+                    name: StaticTestName("second"),
+                    ignore: false,
+                    should_panic: ShouldPanic::No,
+                    serial: true


bjorn3 · 2017-06-20T12:13:10Z

src/libtest/lib.rs

+                TestEvent::TeFilteredOut(n) if n > 0 => panic!("filtered out"),
+                TestEvent::TeTimeout(_) => panic!("timeout"),
+                TestEvent::TeResult(_, ref result, _) if result != &TestResult::TrOk =>
+                    panic!("result not okay"),


Please print the result too. Eg

panic!("result not okay: {:?}", result),

bjorn3 · 2017-06-20T12:13:33Z

src/libtest/lib.rs

+                TestEvent::TeFilteredOut(n) if n > 0 => panic!("filtered out"),
+                TestEvent::TeTimeout(_) => panic!("timeout"),
+                TestEvent::TeResult(_, ref result, _) if result != &TestResult::TrOk =>
+                    panic!("result not okay"),


Same printing thing here.

bors · 2017-06-21T08:47:12Z

☔ The latest upstream changes (presumably #42664) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · 2017-06-25T00:45:44Z

@shivjm I believe this is waiting on some changes from you and a rebase. Let us know if we can help in any way! Feel free to ping me on IRC (simulacrum).

shivjm · 2017-06-25T18:38:48Z

Sorry, been out of town and then busy catching up! I’ve rebased the PR and (I hope) fixed the issues @bjorn3 pointed out (thanks!). Next up, need to implement the feature gate. I’ll ask for help on IRC.

shivjm · 2017-06-25T20:34:34Z

I’ve added the serial_tests feature gate. @est31, does this need an entry under unstable-book as well?

est31 · 2017-06-26T06:11:18Z

does this need an entry under unstable-book as well?

The requirement for an entry has been removed. You can add one (with meaningful content) if you want to, if not, a stub will be generated automatically.

bjorn3

Lgtm when the trailing newline is added.

bjorn3 · 2017-06-26T06:14:23Z

src/test/compile-fail/feature-gate-serial_tests.rs

+#[serial] //~ ERROR serial tests are currently unstable
+fn in_serial() {
+    assert!(true);
+}


Please add a trailing newline.

shivjm · 2017-06-26T06:19:50Z

Done, thanks!

brson · 2017-06-27T21:47:57Z

Thanks for the patch.

This probably needs more discussion before landing. This is a thing that we've declined to do for many years.

brson · 2017-06-27T23:00:26Z

cc @nrc

bors · 2017-06-29T11:18:36Z

☔ The latest upstream changes (presumably #42964) made this pull request unmergeable. Please resolve the merge conflicts.

shivjm · 2017-07-03T12:02:42Z

Is there anything I can do to help?

bors · 2017-07-06T02:39:42Z

☔ The latest upstream changes (presumably #42727) made this pull request unmergeable. Please resolve the merge conflicts.

nrc · 2017-07-10T05:41:17Z

@brson do you have links to previous discussion or could you summarise here please? My opinion (uninformed by previous discussion) is that this is something I've wanted personally and I don't see why we should not support it (as long as it is feature gated etc.) being its own attribute feels kind of wrong (I'd prefer to attach it to #[test], but I suppose that #[should_panic] is a precedent?)

nrc · 2017-07-10T20:20:10Z

Discussed at the dev-tools meeting today, this problem can be solved today using a mutex (I filed #43155 to document that). Given that this solution feels somewhat ad hoc and there is workaround, we would prefer to punt on adding this in favour of custom test runners or some other long-term solution.

Thanks for the PR @shivjm and sorry to close it out after your work.

shivjm · 2017-07-11T09:46:10Z

Alright. Thanks for letting me know.

yshui · 2018-02-21T13:38:33Z

@nrc I don't see why this shouldn't be implemented in the test framework. The mutex work around seems to have some catches, and is non-trivial to get right.

DevQps · 2019-04-05T10:12:20Z

For everyone getting to this point: This solution is also implemented in a separate crate now: serial_test

rust-highfive assigned brson Jun 15, 2017

shepmaster added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 16, 2017

bjorn3 requested changes Jun 20, 2017

View reviewed changes

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 25, 2017

Mark-Simulacrum added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 25, 2017

bjorn3 requested changes Jun 26, 2017

View reviewed changes

bjorn3 approved these changes Jun 26, 2017

View reviewed changes

brson added T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. I-nominated labels Jun 27, 2017

shivjm added 4 commits July 6, 2017 11:24

Add `#[serial]' tests.

38e5839

Test serial tests without needing a hundred of them.

ef5cf97

Use simpler code to run serial tests and remove trailing whitespace.

3fa5e21

Specify serial: false' in make_test' in `compiletest/src/main.rs'.

161d0b5

shivjm added 3 commits July 6, 2017 11:24

Incorporate suggestions from @bjorn3.

9549b7d

Feature-gate #[serial] and add compile-fail test.

f6e5c5e

Add trailing newline to `feature-gate-serial_tests.rs'.

6cf05ab

nrc mentioned this pull request Jul 10, 2017

Document using a Mutex to sequentalise tests #43155

Closed

nrc closed this Jul 10, 2017

nrc mentioned this pull request Jul 10, 2017

Add flag to opt out one test from parallel testing #33519

Closed

yshui unassigned brson Feb 21, 2018

NunoAlexandre mentioned this pull request Feb 10, 2020

Can’t run end-to-end tests concurrently radicle-dev/radicle-registry#214

Closed

4meta5 mentioned this pull request Apr 26, 2020

Double map and plain storage support, introduce macros paritytech/subxt#93

Merged

c3d mentioned this pull request Oct 5, 2020

CI testing failing on [serial(chdir)] kata-containers/kata-containers#856

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ‘#[serial]’ tests (v2) #42684

Add ‘#[serial]’ tests (v2) #42684

shivjm commented Jun 15, 2017

rust-highfive commented Jun 15, 2017

leonardo-m commented Jun 15, 2017

Mark-Simulacrum commented Jun 15, 2017

shivjm commented Jun 15, 2017

bjorn3 left a comment

bjorn3 Jun 20, 2017

bjorn3 Jun 20, 2017

bjorn3 Jun 20, 2017

bjorn3 Jun 20, 2017

bjorn3 Jun 20, 2017

bjorn3 Jun 20, 2017

bors commented Jun 21, 2017

Mark-Simulacrum commented Jun 25, 2017

shivjm commented Jun 25, 2017 •

edited

Loading

shivjm commented Jun 25, 2017

est31 commented Jun 26, 2017

bjorn3 left a comment

bjorn3 Jun 26, 2017

shivjm commented Jun 26, 2017

brson commented Jun 27, 2017

brson commented Jun 27, 2017

bors commented Jun 29, 2017

shivjm commented Jul 3, 2017

bors commented Jul 6, 2017

nrc commented Jul 10, 2017

nrc commented Jul 10, 2017

shivjm commented Jul 11, 2017 via email

yshui commented Feb 21, 2018 •

edited

Loading

DevQps commented Apr 5, 2019

Add ‘#[serial]’ tests (v2) #42684

Add ‘#[serial]’ tests (v2) #42684

Conversation

shivjm commented Jun 15, 2017

rust-highfive commented Jun 15, 2017

leonardo-m commented Jun 15, 2017

Mark-Simulacrum commented Jun 15, 2017

shivjm commented Jun 15, 2017

bjorn3 left a comment

Choose a reason for hiding this comment

bjorn3 Jun 20, 2017

Choose a reason for hiding this comment

bjorn3 Jun 20, 2017

Choose a reason for hiding this comment

bjorn3 Jun 20, 2017

Choose a reason for hiding this comment

bjorn3 Jun 20, 2017

Choose a reason for hiding this comment

bjorn3 Jun 20, 2017

Choose a reason for hiding this comment

bjorn3 Jun 20, 2017

Choose a reason for hiding this comment

bors commented Jun 21, 2017

Mark-Simulacrum commented Jun 25, 2017

shivjm commented Jun 25, 2017 • edited Loading

shivjm commented Jun 25, 2017

est31 commented Jun 26, 2017

bjorn3 left a comment

Choose a reason for hiding this comment

bjorn3 Jun 26, 2017

Choose a reason for hiding this comment

shivjm commented Jun 26, 2017

brson commented Jun 27, 2017

brson commented Jun 27, 2017

bors commented Jun 29, 2017

shivjm commented Jul 3, 2017

bors commented Jul 6, 2017

nrc commented Jul 10, 2017

nrc commented Jul 10, 2017

shivjm commented Jul 11, 2017 via email

yshui commented Feb 21, 2018 • edited Loading

DevQps commented Apr 5, 2019

shivjm commented Jun 25, 2017 •

edited

Loading

yshui commented Feb 21, 2018 •

edited

Loading