gh-108388: Convert test_concurrent_futures to package

[vstinner]

Convert test_concurrent_futures to a package of 7 sub-tests.

Add remote_globals to create_executor_tests()

• Issue: Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

[vstinner]

Timing of running tests: ./python -m test test_concurrent_futures -j0 --slowest on a machine with 12 threads / 6 CPU cores.

• reference: 2 min 43 sec
• PR: 1 min 13 sec

With the PR:

10 slowest tests:
- test_concurrent_futures.test_wait: 1 min 13 sec
- test_concurrent_futures.test_process_pool: 42.6 sec
- test_concurrent_futures.test_shutdown: 20.1 sec
- test_concurrent_futures.test_as_completed: 12.0 sec
- test_concurrent_futures.test_deadlock: 11.1 sec
- test_concurrent_futures.test_future: 3.2 sec
- test_concurrent_futures.test_init: 2.3 sec

[sobolevn]

To fix the CI

[vstinner]

GHA job timings:

• Windows x86: 24 min 37 sec
• Windows x64: 19 min 27 sec
• macOS: 18 min 22 sec
• Ubuntu: 10 min
• Address Sanitizer: 14 min 49 sec

Details.

Windows x86:

10 slowest tests:
- test_importlib: 3 min 39 sec
- test.test_multiprocessing_spawn.test_manager: 2 min 49 sec
- test_compileall: 2 min 47 sec
- test_venv: 2 min 33 sec
- test_tarfile: 2 min 28 sec
- test.test_multiprocessing_spawn.test_processes: 2 min 10 sec
- test_threading: 2 min 2 sec
- test_fstring: 1 min 56 sec
- test_queue: 1 min 52 sec
- test_socket: 1 min 43 sec

Windows x64:

10 slowest tests:
- test.test_multiprocessing_spawn.test_processes: 3 min 18 sec
- test.test_multiprocessing_spawn.test_manager: 2 min 36 sec
- test_regrtest: 2 min 27 sec
- test_importlib: 2 min 6 sec
- test_compileall: 2 min 5 sec
- test_venv: 2 min 2 sec
- test_zipfile: 1 min 57 sec
- test_io: 1 min 47 sec
- test_tarfile: 1 min 45 sec
- test_largefile: 1 min 38 sec

macOS:

10 slowest tests:
- test.test_multiprocessing_spawn.test_processes: 3 min 20 sec
- test_tarfile: 2 min 48 sec
- test.test_multiprocessing_forkserver.test_processes: 2 min 19 sec
- test_pickle: 1 min 58 sec
- test_cppext: 1 min 40 sec
- test_statistics: 1 min 39 sec
- test_compileall: 1 min 25 sec
- test.test_concurrent_futures.test_wait: 1 min 24 sec
- test_subprocess: 1 min 22 sec
- test.test_multiprocessing_spawn.test_misc: 1 min 21 sec

Ubuntu:

10 slowest tests:
- test_gdb: 2 min 23 sec
- test.test_multiprocessing_spawn.test_processes: 1 min 31 sec
- test.test_concurrent_futures.test_wait: 1 min 22 sec
- test_signal: 1 min 2 sec
- test.test_multiprocessing_forkserver.test_processes: 1 min 1 sec
- test_subprocess: 1 min
- test_weakref: 44.6 sec
- test_venv: 44.2 sec
- test_socket: 42.1 sec
- test.test_concurrent_futures.test_process_pool: 41.4 sec

Address Sanitizer:

10 slowest tests:
- test_subprocess: 2 min 11 sec
- test_compileall: 2 min 9 sec
- test_venv: 2 min
- test_gdb: 1 min 39 sec
- test_weakref: 1 min 32 sec
- test_tarfile: 1 min 26 sec
- test_regrtest: 1 min 12 sec
- test_source_encoding: 1 min 11 sec
- test_multiprocessing_main_handling: 1 min 1 sec
- test_decimal: 1 min

[gpshead]

I believe you unintentionally got rid of most of the Threading tests.

(do not merge this even after fixing up this review round's comments, this requires further very close side by side review that Github's UI is incapable of providing as it lacks the ability to track file splits and moves as a diff of what's changed -- I'll need to spend time on the command line in a client looking things over)

Otherwise: Thank you for taking this on! This test suite has needed refactoring into something manageable forever.

[bedevere-bot]

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

[gpshead]

I moved this PR to Draft state to signal that I don't want anyone to think it is ready to merge even if someone approves it until after we've taken a close look to verify that we haven't lost anything in the refactoring.

[vstinner]

I believe you unintentionally got rid of most of the Threading tests.

I compared ./python -m test test_concurrent_futures --list-cases|wc -l with and without my change: I get 248 test cases in both cases.

[vstinner]

For your comments like "ThreadPoolExecutorTest is defined above but missing here" on test_process_pool.py, I suggest you looking at which tests are executed rather than only looking at the code. For example:

$ ./python -m test test_concurrent_futures --list-cases|grep ThreadPoolExecutorTest
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_default_workers
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_executor_map_current_future_cancel
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_free_reference
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_hang_global_shutdown_lock
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_idle_thread_reuse
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_exception
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_submits_without_iteration
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_timeout
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_max_workers_negative
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_no_stale_references
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_saturation
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_shutdown_race_issue12456
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_submit
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_submit_keyword

You can compare between the main branch and my PR: I get the same 15 tests, but with my PR, test names also get an additional test_process_pool. in their name.

[vstinner]

I splitted test_process_pool in two testrs:

• test_thread_pool: 15 tests, takes 10.6 sec
• test_process_pool: 60 tests, takes 32.0 sec

Before, test_process_poll was the sum (75 tests, ~42 sec) 🙂.

[gpshead]

Thanks. confirmed locally as well, all tests are present and accounted for!

[vstinner]

Thanks for the review @gpshead! For sure, there is room for improvement for these tests. I hope that this split will ease future maintenance.

[miss-islington]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[miss-islington]

Sorry, @vstinner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker aa6f787faa4bc45006da4dc2f942fb9b82c98836 3.12

[bedevere-bot]

GH-108443 is a backport of this pull request to the 3.12 branch.

[gpshead]

the merge conflict on this is probably just the list in regrtest runtests.py. retry that automation after the test_multiprocessing one is in 3.12.

[vstinner]

the merge conflict on this is probably just the list in regrtest runtests.py. retry that automation after the test_multiprocessing one is in 3.12.

Right. I backported the change to Python 3.12. There is a minor conflict in libregrtest on:

 SPLITTESTDIRS = {
     "test_asyncio",
+    "test_concurrent_futures",
 }

The backport is in conflict with PR #108401 (split multiprocessing tests) which may be merged first.

[bedevere-app]

GH-109704 is a backport of this pull request to the 3.11 branch.

[vstinner]

Since I was asked to backport all test changes (including refactoring) to stable branches, I backport this change to Python 3.11: GH-109704.