Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-108927: Fix removing testing modules from sys.modules #108952

Merged
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
339f260
gh-108927: Do not remove tested modules from sys.modules
serhiy-storchaka Sep 5, 2023
e52be9b
Update NEWS entry.
serhiy-storchaka Sep 5, 2023
68c014e
Fix unloading the newly imported modules after testing.
serhiy-storchaka Sep 6, 2023
c1ad70c
Merge branch 'main' into regretest-no-remove-test-module
serhiy-storchaka Sep 12, 2023
3f25ab4
Add tests.
serhiy-storchaka Sep 12, 2023
7673248
Update Misc/NEWS.d/next/Tests/2023-09-05-20-46-35.gh-issue-108927.Tpw…
serhiy-storchaka Sep 12, 2023
5df4c8b
Move save/unload modules down in single.py.
serhiy-storchaka Sep 13, 2023
eb7766b
Merge branch 'main' into regretest-no-remove-test-module
serhiy-storchaka Sep 13, 2023
0eae535
Refactoring.
serhiy-storchaka Sep 13, 2023
66437b5
Remove also attributes from parent modules.
serhiy-storchaka Sep 13, 2023
8470b33
Merge branch 'main' into regretest-no-remove-test-module
serhiy-storchaka Oct 14, 2023
544c068
Merge branch 'main' into regretest-no-remove-test-module
serhiy-storchaka Nov 27, 2023
1e7ea34
Merge branch 'main' into regretest-no-remove-test-module
serhiy-storchaka Nov 28, 2023
56621ab
Only unload modules if run tests sequentially.
serhiy-storchaka Nov 28, 2023
ba5597a
Apply suggestions from code review
serhiy-storchaka Dec 1, 2023
d025e22
Merge branch 'main' into regretest-no-remove-test-module
serhiy-storchaka Dec 4, 2023
1aae341
Add a comment.
serhiy-storchaka Dec 4, 2023
017f01d
Test that test modules are really unloaded.
serhiy-storchaka Dec 4, 2023
5086405
Merge remote-tracking branch 'refs/remotes/origin/regretest-no-remove…
serhiy-storchaka Dec 4, 2023
b3ea8fa
Update Lib/test/test_regrtest.py
serhiy-storchaka Dec 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Lib/test/libregrtest/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ def run_tests_sequentially(self, runtests):
else:
tracer = None

save_modules = sys.modules.keys()
save_modules = set(sys.modules)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to move this logic inside libregrtest/single.py, in the run_single_test() function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it, but it is only needed in single-process run. In multi-process run it only adds an overhead. So I moved it back.


jobs = runtests.get_jobs()
if jobs is not None:
Expand All @@ -318,9 +318,9 @@ def run_tests_sequentially(self, runtests):
result = self.run_test(test_name, runtests, tracer)

# Unload the newly imported modules (best effort finalization)
serhiy-storchaka marked this conversation as resolved.
Show resolved Hide resolved
for module in sys.modules.keys():
for module in list(sys.modules):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for module in list(sys.modules):
for module in sys.modules:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No it does not work, because the dict is modified during iteration.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this code. Why do we have to unload test modules?

If it's important to unload test modules, I would suggest to move this code inside single.py, to make a more consistent behavior for the different ways to run tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you ask why the code of libregrtest is so complicated? 🤷‍♂️

I think that it only makes sense in the single-process run. As well as removing the testing module before reloading in _load_run_test(). In the multi-process run it is only a waste of time. As well as significant part of save_env (except cleaning up external resources like filesystem).

if module not in save_modules and module.startswith("test."):
support.unload(module)
sys.modules.pop(module, None)

if result.must_stop(self.fail_fast, self.fail_env_changed):
break
Expand Down
4 changes: 0 additions & 4 deletions Lib/test/libregrtest/single.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,6 @@ def _load_run_test(result: TestResult, runtests: RunTests) -> None:
# Load the test module and run the tests.
test_name = result.test_name
module_name = abs_module_name(test_name, runtests.test_dir)

# Remove the module from sys.module to reload it if it was already imported
sys.modules.pop(module_name, None)

test_mod = importlib.import_module(module_name)

if hasattr(test_mod, "test_main"):
Expand Down
6 changes: 6 additions & 0 deletions Lib/test/regrtestdata/import_from_tests/test_regrtest_a.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import unittest
import test_regrtest_b.util

class Test(unittest.TestCase):
def test(self):
test_regrtest_b.util
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import unittest

class Test(unittest.TestCase):
def test(self):
pass
Empty file.
6 changes: 6 additions & 0 deletions Lib/test/regrtestdata/import_from_tests/test_regrtest_c.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import unittest
import test_regrtest_b.util

class Test(unittest.TestCase):
def test(self):
test_regrtest_b.util
8 changes: 8 additions & 0 deletions Lib/test/test_regrtest.py
Original file line number Diff line number Diff line change
Expand Up @@ -1866,6 +1866,14 @@ def test_random_seed(self):
def test_random_seed_workers(self):
self._check_random_seed(run_workers=True)

def test_import_from_tests(self):
serhiy-storchaka marked this conversation as resolved.
Show resolved Hide resolved
testdir = os.path.join(os.path.dirname(__file__),
'regrtestdata', 'import_from_tests')
tests = [f'test_regrtest_{name}' for name in ('a', 'b', 'c')]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind to add a comment to explain the purpose of these test? By reading the code, it's unclear to me. Explain that the test execution order matters for this test.

Does the test fails if a regression is introduced? Maybe add a test_regrtest_d test checking if test_regrtest_a, test_regrtest_b and test_regrtest_c are not loaded?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

args = ['-Wd', '-E', '-bb', '-m', 'test', '--testdir=%s' % testdir, *tests]
output = self.run_python(args)
self.check_executed_tests(output, tests, stats=3)


class TestUtils(unittest.TestCase):
def test_format_duration(self):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Fixed order dendence in running tests in the same process
serhiy-storchaka marked this conversation as resolved.
Show resolved Hide resolved
when a test that has submodules (e.g. test_importlib) follows a test that
imports its submodule (e.g. test_importlib.util) and precedes a test
(e.g. test_unittest or test_compileall) that uses that submodule.
Loading