Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

vstinner · 2023-08-23T23:05:39Z

The slowest tests of the Python test suite are:

test_concurrent_futures
test_multiprocessing_spawn
test_peg_generator
test_tools.test_freeze tests

They each tend take multiple minutes to run. The bulk of our others take <10 seconds. They occupy a lot of wall time as a long tail in a normal make test or regrtest run on a typical parallel run multi-core systems.

This bug originally proposed to skip them unless the "cpu" resource is enabled (it's disabled by default). That was deemed appropriate for test_peg_generator and test_tools.test_freeze which are either not platform specific or rarely needed in CI. (PR #108386) - This PR reduced the total test duration between 3 and 5 minutes.

Linked PRs

The text was updated successfully, but these errors were encountered:

The test_concurrent_futures and test_multiprocessing_spawn tests now require the 'cpu' resource. Skip these tests unless the 'cpu' resource is enabled (it is disabled by default). test_concurrent_futures is no longer skipped if Python is built with ASAN or MSAN sanitizer.

The Python test suite (regrtest) now runs the 20 slowest tests first and then other tests, to better use all available CPUs when running tests in parallel.

Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio".

Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn in packages made of 4 sub-tests (processes, threads, manager and misc).

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn in test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.

Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code.

Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code. (cherry picked from commit 174e9da) Co-authored-by: Victor Stinner <vstinner@python.org>

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.

* Split test_concurrent_futures.py into multiple files in a new Lib/test/test_concurrent_futures/ package. * Add remote_globals to create_executor_tests()

terryjreedy · 2023-08-24T04:19:20Z

I have previously requested that the slowest tests not restricted to -cpu be split into pieces that can run in parallel when -j is used. Thank you for doing this.

Convert test_concurrent_futures to a package of 7 sub-tests. Add remote_globals to create_executor_tests()

hugovk · 2023-08-24T05:09:32Z

The slowest tests of the Python test suite are:

test_concurrent_futures

test_multiprocessing_spawn

test_peg_generator

test_tools.test_freeze tests

How long do they usually take?

…108397) gh-108388: regrtest splits test_asyncio package (GH-108393) Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code. (cherry picked from commit 174e9da) Co-authored-by: Victor Stinner <vstinner@python.org>

Convert test_concurrent_futures to a package of sub-tests.

gpshead · 2023-08-24T17:33:43Z

I edited the title and updated the initial comment since this and the related PRs shifted from the original opening statement. :)

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

…08401) Convert test_concurrent_futures to a package of sub-tests. (cherry picked from commit aa6f787)

sobolevn · 2023-08-25T07:00:09Z

I experience new failures of test_concurrent_futures on Windows on my unrelated PR: https://github.com/python/cpython/actions/runs/5972558288/job/16203298171?pr=108456

Failing tests:

     test.test_concurrent_futures.test_deadlock
     test.test_concurrent_futures.test_shutdown

vstinner · 2023-08-25T09:35:34Z

I experience new failures of test_concurrent_futures on Windows on my unrelated PR

Sadly, the issue is known for at least one month: see issue #107219.

Logs:

  File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures\test_deadlock.py", line 236 in test_crash_big_data

It's this test which hangs sometimes on Windows.

gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

#108443) gh-108388: Convert test_concurrent_futures to package (#108401) Convert test_concurrent_futures to a package of sub-tests. (cherry picked from commit aa6f787)

Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code. (cherry picked from commit 174e9da)

…108820) * Revert "[3.11] gh-101634: regrtest reports decoding error as failed test (#106169) (#106175)" This reverts commit d5418e9. * Revert "[3.11] bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (GH-30895) (GH-103342)" This reverts commit ecb09a8. * Revert "gh-95027: Fix regrtest stdout encoding on Windows (GH-98492)" This reverts commit b2aa28e. * Revert "[3.11] gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253) (GH-94408)" This reverts commit 0122ab2. * Revert "Run Tools/scripts/reindent.py (GH-94225)" This reverts commit f0f3a42. * Revert "gh-94052: Don't re-run failed tests with --python option (GH-94054)" This reverts commit 1347607. * Revert "[3.11] gh-84461: Fix Emscripten umask and permission issues (GH-94002) (GH-94006)" This reverts commit 1073184. * gh-93353: regrtest checks for leaked temporary files (#93776) When running tests with -jN, create a temporary directory per process and mark a test as "environment changed" if a test leaks a temporary file or directory. (cherry picked from commit e566ce5) * gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813) (cherry picked from commit 36934a1) * gh-93353: regrtest supports checking tmp files with -j2 (#93909) regrtest now also implements checking for leaked temporary files and directories when using -jN for N >= 2. Use tempfile.mkdtemp() to create the temporary directory. Skip this check on WASI. (cherry picked from commit 4f85cec) * gh-84461: Fix Emscripten umask and permission issues (GH-94002) - Emscripten's default umask is too strict, see emscripten-core/emscripten#17269 - getuid/getgid and geteuid/getegid are stubs that always return 0 (root). Disable effective uid/gid syscalls and fix tests that use chmod() current user. - Cannot drop X bit from directory. (cherry picked from commit 2702e40) * gh-94052: Don't re-run failed tests with --python option (#94054) (cherry picked from commit 0ff7b99) * Run Tools/scripts/reindent.py (#94225) Reindent files which were not properly formatted (PEP 8: 4 spaces). Remove also some trailing spaces. (cherry picked from commit e87ada4) * gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253) Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit 199ba23) * gh-96465: Clear fractions hash lru_cache under refleak testing (GH-96689) Automerge-Triggered-By: GH:zware (cherry picked from commit 9c8f379) * gh-95027: Fix regrtest stdout encoding on Windows (#98492) On Windows, when the Python test suite is run with the -jN option, the ANSI code page is now used as the encoding for the stdout temporary file, rather than using UTF-8 which can lead to decoding errors. (cherry picked from commit ec1f6f5) * gh-98903: Test suite fails with exit code 4 if no tests ran (#98904) The Python test suite now fails wit exit code 4 if no tests ran. It should help detecting typos in test names and test methods. * Add "EXITCODE_" constants to Lib/test/libregrtest/main.py. * Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN" (cherry picked from commit c76db37) * gh-100086: Add build info to test.libregrtest (#100093) The Python test runner (libregrtest) now logs Python build information like "debug" vs "release" build, or LTO and PGO optimizations. (cherry picked from commit 3c89202) * bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> (cherry picked from commit 9953860) * gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927) This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel. After porting we can see the direct impact on a multicore system. Without this change: Running make test is 5 min 26 seconds With this change: Running make test takes 3 min 39 seconds That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%. The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant. Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests. --------- Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC] (cherry picked from commit 9e011e7) * Display the sanitizer config in the regrtest header. (#105301) Display the sanitizers present in libregrtest. Having this in the CI output for tests with the relevant environment variable displayed will help make it easier to do what we need to create an equivalent local test run. (cherry picked from commit 852348a) * gh-101634: regrtest reports decoding error as failed test (#106169) When running the Python test suite with -jN option, if a worker stdout cannot be decoded from the locale encoding report a failed testn so the exitcode is non-zero. (cherry picked from commit 2ac3eec) * gh-108223: test.pythoninfo and libregrtest log Py_NOGIL (#108238) Enable with --disable-gil --without-pydebug: $ make pythoninfo|grep NOGIL sysconfig[Py_NOGIL]: 1 $ ./python -m test ... == Python build: nogil debug ... (cherry picked from commit 5afe0c1) * gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289) * Cleanup libregrtest code logging ASAN_OPTIONS. * Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS". (cherry picked from commit 3a1ac87) * gh-108388: regrtest splits test_asyncio package (#108393) Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code. (cherry picked from commit 174e9da) * regrtest computes statistics (#108793) test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> (cherry picked from commit d4e534c) * gh-108822: Add Changelog entry for regrtest statistics (#108821) --------- Co-authored-by: Christian Heimes <christian@python.org> Co-authored-by: Zachary Ware <zach@python.org> Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Joshua Herman <zitterbewegung@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>

vstinner · 2023-09-13T04:28:51Z

This feature has been implemented in the main branch, and backported to the 3.12 branch. I don't think that it's worth it to backport it to the 3.11 branch.

I splitted these test packages:

test_concurrent_futures
test_multiprocessing_fork
test_multiprocessing_forkserver
test.test_multiprocessing_spawn

Example:

$ ./python -m test test_concurrent_futures --list-tests
test_concurrent_futures.test_as_completed
test_concurrent_futures.test_deadlock
test_concurrent_futures.test_future
test_concurrent_futures.test_init
test_concurrent_futures.test_process_pool
test_concurrent_futures.test_shutdown
test_concurrent_futures.test_thread_pool
test_concurrent_futures.test_wait

I close my issue.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <vstinner@python.org>

#109704) * gh-108388: Convert test_concurrent_futures to package (#108401) Convert test_concurrent_futures to a package of sub-tests. (cherry picked from commit aa6f787) Notes on backport to 3.11: * AsCompletedTests: Revert test_future_times_out() => test_zero_timeout() * Restore TODO comment * ThreadPoolExecutorTest.test_hang_global_shutdown_lock(): add @support.requires_resource('cpu').

vstinner added the type-bug An unexpected behavior, bug, or error label Aug 23, 2023

bedevere-bot mentioned this issue Aug 23, 2023

gh-108388: test_concurrent_futures requires cpu #108389

Closed

bedevere-bot mentioned this issue Aug 24, 2023

gh-108388: regrtest runs slowest tests first #108391

Closed

bedevere-bot mentioned this issue Aug 24, 2023

gh-108388: regrtest splits test_asyncio package #108393

Merged

bedevere-bot mentioned this issue Aug 24, 2023

gh-108388: Split test_multiprocessing_spawn #108396

Merged

bedevere-bot mentioned this issue Aug 24, 2023

[3.12] gh-108388: regrtest splits test_asyncio package (GH-108393) #108397

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Aug 24, 2023

pythongh-108388: Convert test_concurrent_futures to package

56d01d0

Convert test_concurrent_futures to a package of 7 sub-tests. Add remote_globals to create_executor_tests()

bedevere-bot mentioned this issue Aug 24, 2023

gh-108388: Convert test_concurrent_futures to package #108401

Merged

AlexWaygood added the tests Tests in the Lib/test dir label Aug 24, 2023

sobolevn mentioned this issue Aug 24, 2023

Mark slow test methods with @requires_resource('cpu') #108416

Closed

vstinner added a commit that referenced this issue Aug 24, 2023

gh-108388: Convert test_concurrent_futures to package (#108401)

aa6f787

Convert test_concurrent_futures to a package of sub-tests.

gpshead changed the title ~~Skip slowest tests unless the "cpu" test resource is enabled (disabled by default)~~ Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. Aug 24, 2023

bedevere-bot mentioned this issue Aug 24, 2023

[3.12] gh-108388: Split test_multiprocessing_spawn (GH-108396) #108442

Merged

bedevere-bot mentioned this issue Aug 24, 2023

[3.12] gh-108388: Convert test_concurrent_futures to package (#108401) #108443

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Aug 24, 2023

pythongh-108388: Convert test_concurrent_futures to package (python#1…

d377273

…08401) Convert test_concurrent_futures to a package of sub-tests. (cherry picked from commit aa6f787)

gpshead added 3.12 bugs and security fixes build The build process and cross-build and removed type-bug An unexpected behavior, bug, or error labels Aug 24, 2023

vstinner mentioned this issue Sep 2, 2023

[3.11] gh-108822: Backport libregrtest changes from the main branch #108820

Merged

vstinner closed this as completed Sep 13, 2023

bedevere-app bot mentioned this issue Sep 21, 2023

[3.11] gh-108388: Split test_multiprocessing_spawn (GH-108396) #109688

Merged

bedevere-app bot mentioned this issue Sep 22, 2023

[3.11] gh-108388: Convert test_concurrent_futures to package (#108401) #109704

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

vstinner commented Aug 23, 2023 •

edited by bedevere-app bot

Loading

terryjreedy commented Aug 24, 2023

hugovk commented Aug 24, 2023

gpshead commented Aug 24, 2023

sobolevn commented Aug 25, 2023 •

edited

Loading

vstinner commented Aug 25, 2023

vstinner commented Sep 13, 2023

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Comments

vstinner commented Aug 23, 2023 • edited by bedevere-app bot Loading

Linked PRs

terryjreedy commented Aug 24, 2023

hugovk commented Aug 24, 2023

gpshead commented Aug 24, 2023

sobolevn commented Aug 25, 2023 • edited Loading

vstinner commented Aug 25, 2023

vstinner commented Sep 13, 2023

vstinner commented Aug 23, 2023 •

edited by bedevere-app bot

Loading

sobolevn commented Aug 25, 2023 •

edited

Loading