Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') #108480

Merged
merged 4 commits into from
Sep 5, 2023
Merged

gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') #108480

merged 4 commits into from
Sep 5, 2023

Conversation

serhiy-storchaka
Copy link
Member

@serhiy-storchaka serhiy-storchaka commented Aug 25, 2023 •
edited by bedevere-bot
Loading

@serhiy-storchaka serhiy-storchaka added tests Tests in the Lib/test dir skip news needs backport to 3.11 only security fixes needs backport to 3.12 bug and security fixes labels Aug 25, 2023
@serhiy-storchaka serhiy-storchaka requested review from a team and vsajip as code owners August 25, 2023 11:33
@bedevere-bot bedevere-bot mentioned this pull request Aug 25, 2023
Closed
@serhiy-storchaka serhiy-storchaka removed request for a team and vsajip August 25, 2023 12:47
@serhiy-storchaka serhiy-storchaka mentioned this pull request Sep 2, 2023
Merged
@serhiy-storchaka
Copy link
Member Author

These are mostly tests which contain several sleep()s or call a function which expected to fail by timeout. There are few tests which depend on slow external network service.

@serhiy-storchaka
Copy link
Member Author 

$ ./python -m test -uall,-cpu -j4
...
Total duration: 7 min 27 sec
Total tests: run=42,127 skipped=1,077

$ ./python -m test -uall,-cpu,-walltime -j4
...
Total duration: 6 min 26 sec
Total tests: run=42,127 skipped=1,139

@serhiy-storchaka serhiy-storchaka mentioned this pull request Sep 5, 2023
Open
@serhiy-storchaka serhiy-storchaka merged commit 1e0d627 into python:main Sep 5, 2023
15 checks passed
@miss-islington
Copy link
Contributor

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

@serhiy-storchaka serhiy-storchaka deleted the mark-slow-tests-walltime branch September 5, 2023 14:56
@miss-islington
Copy link
Contributor

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 1e0d62793a84001e92f1c80b511d3a212b435acc 3.11

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 5, 2023
@serhiy-storchaka @miss-islington
pythongh-108416: Mark slow but not CPU bound test methods with requir…
7cd8441 
…es_resource('walltime') (pythonGH-108480)

(cherry picked from commit 1e0d627)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot
Copy link

GH-108923 is a backport of this pull request to the 3.12 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.12 bug and security fixes label Sep 5, 2023
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this pull request Sep 5, 2023
@serhiy-storchaka
[3.11] pythongh-108416: Mark slow but not CPU bound test methods with…
65b5218 
… requires_resource('walltime') (pythonGH-108480).

(cherry picked from commit 1e0d627)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot
Copy link

GH-108924 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Sep 5, 2023
serhiy-storchaka added a commit that referenced this pull request Sep 5, 2023
@serhiy-storchaka
[3.11] gh-108416: Mark slow but not CPU bound test methods with requi…
cf19e8e 
…res_resource('walltime') (GH-108480) (GH-108924)

(cherry picked from commit 1e0d627)
vstinner pushed a commit to vstinner/cpython that referenced this pull request Sep 5, 2023
@serhiy-storchaka @vstinner
pythongh-108416: Mark slow but not CPU bound test methods with requir…
2cf86ef 
…es_resource('walltime') (pythonGH-108480)

(cherry picked from commit 1e0d627)
@vstinner vstinner mentioned this pull request Sep 5, 2023
Merged
Yhg1s pushed a commit that referenced this pull request Sep 8, 2023
@vstinner @serhiy-storchaka
[3.12] gh-108834: Sync libregrtest with the main branch (#108966)
0855b2c 
* gh-108834: regrtest reruns failed tests in subprocesses (#108839)

When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.

(cherry picked from commit 31c2945)

* gh-108834: regrtest --fail-rerun exits with code 5 (#108896)

When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").

(cherry picked from commit 1170d5a)

* gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480)

(cherry picked from commit 1e0d627)

* Manually sync Lib/test/libregrtest/ from main

---------

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@serhiy-storchaka serhiy-storchaka removed their assignment Sep 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gpshead gpshead gpshead approved these changes

@ambv ambv Awaiting requested review from ambv

Assignees
No one assigned
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@serhiy-storchaka @miss-islington @bedevere-bot @gpshead