bpo-46542: test_lib2to3 uses support.infinite_recursion() #31035

vstinner · 2022-01-31T16:35:24Z

Fix a Python crash in test_lib2to3 when using Python built in debug
mode: limit the recursion limit.

The test_all_project_files() test of test_lib2to3 now uses the
test.support.infinite_recursion() context manager when processing the
infinite_recursion.py file to prevent a crash when Python is built in
debug mode.

The two test_all_project_files() tests now use subTest() and log the
refactored/parsed filename (if test_lib2to3 is run in verbose mode).

https://bugs.python.org/issue46542

Fix a Python crash in test_lib2to3 when using Python built in debug mode: limit the recursion limit. The test_all_project_files() test of test_lib2to3 now uses the test.support.infinite_recursion() context manager when processing the infinite_recursion.py file to prevent a crash when Python is built in debug mode. The two test_all_project_files() tests now use subTest() and log the refactored/parsed filename (if test_lib2to3 is run in verbose mode).

vstinner · 2022-01-31T16:36:48Z

cc @pablogsal @ambv @markshannon

vstinner · 2022-01-31T16:40:55Z

I built Python in debug mode with clang:

$ ./configure --with-pydebug CC=clang
$ grep -- -O Makefile
OPT=		-g -O0 -Wall
$ make

clang uses -O0. Stack usage:

$ ./python
Python 3.11.0a4+ (heads/lib2to3_infinite:01e926a388, Jan 31 2022, 17:21:47) [Clang 13.0.0 (Fedora 13.0.0-3.fc35)] on linux
>>> import _testcapi
>>> def f(): yield _testcapi.stack_pointer()
... 
>>> _testcapi.stack_pointer() - next(f())
9104

Without this change (current main branch, at commit 7685693), test_lib2to3 does crash :-(

$ ./python -m test -u all test_lib2to3 -v
(...)
test_all_project_files (lib2to3.tests.test_all_fixers.Test_all) ... 
Fatal Python error: Segmentation fault

Current thread 0x00007f455c394740 (most recent call first):
  File "/home/vstinner/python/main/Lib/lib2to3/pytree.py", line 845 in generate_matches
  File "/home/vstinner/python/main/Lib/lib2to3/pytree.py", line 785 in _recursive_matches
  File "/home/vstinner/python/main/Lib/lib2to3/pytree.py", line 786 in _recursive_matches
(...)

With this PR, it no longer crash:

$ ./python -m test -u all test_lib2to3 -v
(...)
Tests result: SUCCESS

vstinner · 2022-01-31T16:44:39Z

Lib/lib2to3/tests/test_all_fixers.py: if the recursion limit is too low, self.refactor.refactor_file(filepath) raises RecursionError. If it's too high, Python does crash. I chose an absolute limite of 150 which is a compromise.

By the way, test.support.infinite_recursion() could be enhanced to use _testinternalcapi.get_recursion_depth() to implement a relative limit, raher than an absolute limit, but that's a different issue ;-) The problem is that the function parameter is an absolute limit.

Lib/lib2to3/tests/data/infinite_recursion.py

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

miss-islington · 2022-01-31T17:46:12Z

Thanks @vstinner for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10.
🐍🍒⛏🤖

…1035) * bpo-46542: test_lib2to3 uses support.infinite_recursion() Fix a Python crash in test_lib2to3 when using Python built in debug mode: limit the recursion limit. The test_all_project_files() test of test_lib2to3 now uses the test.support.infinite_recursion() context manager when processing the infinite_recursion.py file to prevent a crash when Python is built in debug mode. The two test_all_project_files() tests now use subTest() and log the refactored/parsed filename (if test_lib2to3 is run in verbose mode). * Update Lib/lib2to3/tests/data/infinite_recursion.py Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> (cherry picked from commit ee0ac32) Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-bot · 2022-01-31T17:47:04Z

GH-31036 is a backport of this pull request to the 3.10 branch.

…1035) * bpo-46542: test_lib2to3 uses support.infinite_recursion() Fix a Python crash in test_lib2to3 when using Python built in debug mode: limit the recursion limit. The test_all_project_files() test of test_lib2to3 now uses the test.support.infinite_recursion() context manager when processing the infinite_recursion.py file to prevent a crash when Python is built in debug mode. The two test_all_project_files() tests now use subTest() and log the refactored/parsed filename (if test_lib2to3 is run in verbose mode). * Update Lib/lib2to3/tests/data/infinite_recursion.py Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> (cherry picked from commit ee0ac32) Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-bot · 2022-01-31T17:47:08Z

GH-31037 is a backport of this pull request to the 3.9 branch.

* bpo-46542: test_lib2to3 uses support.infinite_recursion() Fix a Python crash in test_lib2to3 when using Python built in debug mode: limit the recursion limit. The test_all_project_files() test of test_lib2to3 now uses the test.support.infinite_recursion() context manager when processing the infinite_recursion.py file to prevent a crash when Python is built in debug mode. The two test_all_project_files() tests now use subTest() and log the refactored/parsed filename (if test_lib2to3 is run in verbose mode). * Update Lib/lib2to3/tests/data/infinite_recursion.py Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> (cherry picked from commit ee0ac32) Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-bot · 2022-01-31T22:03:07Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Shared 3.9 has failed when building commit 3046386.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/151/builds/439) and take a look at the build logs.
Check if the failure is related to this commit (3046386) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/151/builds/439

Failed tests:

test_smtpnet
test_unicodedata

Failed subtests:

test_connect - test.test_smtpnet.SmtpSSLTest
test_connect_default_port - test.test_smtpnet.SmtpSSLTest
test_normalization - test.test_unicodedata.NormalizationTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

405 tests OK.

10 slowest tests:

test_codecmaps_jp: 9 min 16 sec
test_tokenize: 8 min 26 sec
test_multiprocessing_spawn: 7 min 41 sec
test_unparse: 6 min 23 sec
test_codecmaps_cn: 6 min 3 sec
test_concurrent_futures: 5 min 2 sec
test_multiprocessing_forkserver: 4 min 28 sec
test_unicodedata: 4 min 17 sec
test_httplib: 4 min 16 sec
test_gdb: 4 min 13 sec

2 tests failed:
test_smtpnet test_unicodedata

18 tests skipped:
test_dbm_gnu test_devpoll test_epoll test_idle test_ioctl
test_msilib test_spwd test_startfile test_tcl test_tix test_tk
test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio
test_winreg test_winsound test_zipfile64

2 re-run tests:
test_smtpnet test_unicodedata

Total duration: 44 min 42 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_unicodedata.py", line 330, in test_normalization
    testdata = open_urlresource(TESTDATAURL, encoding="utf-8",
http.client.IncompleteRead: IncompleteRead(0 bytes read)


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
socket.gaierror: [Errno 8] Name does not resolve


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_unicodedata.py", line 336, in test_normalization
    self.fail(f"Could not retrieve {TESTDATAURL}")
AssertionError: Could not retrieve http://www.pythontest.net/unicode/13.0.0/NormalizationTest.txt


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_smtpnet.py", line 59, in test_connect_default_port
    server = smtplib.SMTP_SSL(self.testServer)
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 1050, in __init__
    SMTP.__init__(self, host, port, local_hostname, timeout,
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 343, in connect
    (code, msg) = self.getreply()
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 405, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/http/client.py", line 597, in _readinto_chunked
    n = self._safe_readinto(mvb)
http.client.IncompleteRead: IncompleteRead(296 bytes read, 1795 more expected)


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_unicodedata.py", line 330, in test_normalization
    testdata = open_urlresource(TESTDATAURL, encoding="utf-8",
urllib.error.URLError: <urlopen error [Errno 8] Name does not resolve>


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_smtpnet.py", line 52, in test_connect
    server = smtplib.SMTP_SSL(self.testServer, self.remotePort)
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 1050, in __init__
    SMTP.__init__(self, host, port, local_hostname, timeout,
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 255, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 343, in connect
    (code, msg) = self.getreply()
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/smtplib.py", line 405, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed

JelleZijlstra · 2022-01-31T22:06:22Z

Looks like DNS is broken on the buildbot. Unlikely to be related to this PR.

…1035) * bpo-46542: test_lib2to3 uses support.infinite_recursion() Fix a Python crash in test_lib2to3 when using Python built in debug mode: limit the recursion limit. The test_all_project_files() test of test_lib2to3 now uses the test.support.infinite_recursion() context manager when processing the infinite_recursion.py file to prevent a crash when Python is built in debug mode. The two test_all_project_files() tests now use subTest() and log the refactored/parsed filename (if test_lib2to3 is run in verbose mode). * Update Lib/lib2to3/tests/data/infinite_recursion.py Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> (cherry picked from commit ee0ac32) Co-authored-by: Victor Stinner <vstinner@python.org>

the-knights-who-say-ni added the CLA signed label Jan 31, 2022

bedevere-bot added the awaiting core review label Jan 31, 2022

vstinner added needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels Jan 31, 2022

vstinner mentioned this pull request Jan 31, 2022

bpo-46329: Split calls into precall and call instructions. #30855

Merged

ambv approved these changes Jan 31, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jan 31, 2022

JelleZijlstra reviewed Jan 31, 2022

View reviewed changes

Lib/lib2to3/tests/data/infinite_recursion.py Outdated Show resolved Hide resolved

Update Lib/lib2to3/tests/data/infinite_recursion.py

25224fd

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

ambv merged commit ee0ac32 into python:main Jan 31, 2022

bedevere-bot removed the awaiting merge label Jan 31, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Jan 31, 2022

bedevere-bot removed the needs backport to 3.9 only security fixes label Jan 31, 2022

markshannon mentioned this pull request Feb 1, 2022

bpo-46329: Reduce default recursion to 800 if Py_DEBUG is enabled. #31033

Closed

vstinner deleted the lib2to3_infinite branch December 13, 2022 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-46542: test_lib2to3 uses support.infinite_recursion() #31035

bpo-46542: test_lib2to3 uses support.infinite_recursion() #31035

vstinner commented Jan 31, 2022 •

edited by bedevere-bot

Loading

vstinner commented Jan 31, 2022

vstinner commented Jan 31, 2022

vstinner commented Jan 31, 2022

miss-islington commented Jan 31, 2022

bedevere-bot commented Jan 31, 2022

bedevere-bot commented Jan 31, 2022

bedevere-bot commented Jan 31, 2022

JelleZijlstra commented Jan 31, 2022

bpo-46542: test_lib2to3 uses support.infinite_recursion() #31035

bpo-46542: test_lib2to3 uses support.infinite_recursion() #31035

Conversation

vstinner commented Jan 31, 2022 • edited by bedevere-bot Loading

vstinner commented Jan 31, 2022

vstinner commented Jan 31, 2022

vstinner commented Jan 31, 2022

miss-islington commented Jan 31, 2022

bedevere-bot commented Jan 31, 2022

bedevere-bot commented Jan 31, 2022

bedevere-bot commented Jan 31, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

JelleZijlstra commented Jan 31, 2022

vstinner commented Jan 31, 2022 •

edited by bedevere-bot

Loading