gh-104372: Cleanup _posixsubprocess `make_inheritable` for async signal safety #104518

gpshead · 2023-05-16T00:08:47Z

We move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET* from the post-fork/vfork child process.

Much of this was long overdue. We shouldn't have been using PyTuple and PyLong APIs within all of the lower level functions anyways.

Issue: subprocess can block all Python threads when using vfork() until the child process exec() succeeds or fails. #104372

We move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process.

gpshead · 2023-05-16T00:11:27Z

This is a code cleanup that #104515 mentions as desirable so that a nicer PR can be made for the larger issue.

bedevere-bot · 2023-05-16T03:32:38Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit adae5af 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2023-05-16T03:32:39Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit adae5af 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

Eclips4 · 2023-05-16T06:34:18Z

Fails in test_gzip, test_httpservers, test_nntplib, test_tarfile, test_xmlrpc related to isolation of io module.
(see #104510 for more details)

bedevere-bot · 2023-05-16T19:35:08Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit bfc9d9e 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2023-05-16T19:35:14Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit bfc9d9e 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

bedevere-bot · 2023-05-16T19:35:16Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit bfc9d9e 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

gpshead · 2023-05-17T07:57:26Z

!buildbot root

bedevere-bot · 2023-05-17T07:57:28Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit e073dde 🤖

The command will test the builders whose names match following regular expression: root

The builders matched are:

gpshead · 2023-05-17T07:57:48Z

!buildbot Debian root

bedevere-bot · 2023-05-17T07:57:51Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit e073dde 🤖

The command will test the builders whose names match following regular expression: Debian root

The builders matched are:

gpshead · 2023-05-17T07:58:04Z

!buildbot AMD64 Debian root

bedevere-bot · 2023-05-17T07:58:06Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit e073dde 🤖

The command will test the builders whose names match following regular expression: AMD64 Debian root

The builders matched are:

AMD64 Debian root PR

gpshead · 2023-05-17T08:03:44Z

!buildbot AMD64 Debian root

bedevere-bot · 2023-05-17T08:03:46Z

🤖 New build scheduled with the buildbot fleet by @gpshead for commit cd3108c 🤖

The command will test the builders whose names match following regular expression: AMD64 Debian root

The builders matched are:

AMD64 Debian root PR

miss-islington · 2023-05-17T16:00:04Z

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington · 2023-05-17T16:00:06Z

Sorry @gpshead, I had trouble checking out the 3.11 backport branch.
Please retry by removing and re-adding the "needs backport to 3.11" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker c649df63e0d052044a4660101d5769ff46ae9234 3.11

bedevere-bot · 2023-05-17T16:22:52Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora 3.x has failed when building commit c649df6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/223/builds/3795) and take a look at the build logs.
Check if the failure is related to this commit (c649df6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/223/builds/3795

Failed tests:

test_tools

Failed subtests:

test_freeze_simple_script - test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

444 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 46 sec
test_multiprocessing_spawn: 2 min 14 sec
test_math: 2 min 12 sec
test_tools: 2 min
test_multiprocessing_forkserver: 1 min 25 sec
test_multiprocessing_fork: 1 min 16 sec
test_tokenize: 1 min 13 sec
test_signal: 1 min 3 sec
test_gdb: 1 min 2 sec
test_socket: 52.4 sec

1 test failed:
test_tools

20 tests skipped:
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test_devpoll test_ioctl
test_kqueue test_launcher test_msilib test_nis test_peg_generator
test_perf_profiler test_readline test_startfile test_tix
test_tkinter test_ttk test_winconsoleio test_winreg test_winsound
test_wmi test_zipfile64

1 re-run test:
test_tools

Total duration: 7 min 49 sec

Click to see traceback logs

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/test_python_3ag2_1jl/tmpu9ksrovt/cpython/Lib/sysconfig.py", line 740, in get_config_var
    return get_config_vars().get(name)
           ^^^^^^^^^^^^^^^^^
  File "/tmp/test_python_3ag2_1jl/tmpu9ksrovt/cpython/Lib/sysconfig.py", line 723, in get_config_vars
    _init_config_vars()
  File "/tmp/test_python_3ag2_1jl/tmpu9ksrovt/cpython/Lib/sysconfig.py", line 670, in _init_config_vars
    _init_posix(_CONFIG_VARS)
  File "/tmp/test_python_3ag2_1jl/tmpu9ksrovt/cpython/Lib/sysconfig.py", line 536, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata_d_linux_s390x-linux-gnu'


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/test/test_tools/test_freeze.py", line 27, in test_freeze_simple_script
    outdir, scriptfile, python = helper.prepare(script, outdir)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Tools/freeze/test/freeze.py", line 184, in prepare
    _run_quiet([MAKE, '-C', builddir, 'install'])
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Tools/freeze/test/freeze.py", line 25, in _run_quiet
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/test_python_3ag2_1jl/tmpu9ksrovt/python-build', 'install']' returned non-zero exit status 2.


Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/tmpgtasm7e9/cpython/Lib/sysconfig.py", line 740, in get_config_var
    return get_config_vars().get(name)
           ^^^^^^^^^^^^^^^^^
  File "/tmp/tmpgtasm7e9/cpython/Lib/sysconfig.py", line 723, in get_config_vars
    _init_config_vars()
  File "/tmp/tmpgtasm7e9/cpython/Lib/sysconfig.py", line 670, in _init_config_vars
    _init_posix(_CONFIG_VARS)
  File "/tmp/tmpgtasm7e9/cpython/Lib/sysconfig.py", line 536, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata_d_linux_s390x-linux-gnu'


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/test/test_tools/test_freeze.py", line 27, in test_freeze_simple_script
    outdir, scriptfile, python = helper.prepare(script, outdir)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Tools/freeze/test/freeze.py", line 184, in prepare
    _run_quiet([MAKE, '-C', builddir, 'install'])
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Tools/freeze/test/freeze.py", line 25, in _run_quiet
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/tmpgtasm7e9/python-build', 'install']' returned non-zero exit status 2.

bedevere-bot · 2023-05-17T16:32:49Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Clang 3.x has failed when building commit c649df6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/3/builds/3921) and take a look at the build logs.
Check if the failure is related to this commit (c649df6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/3/builds/3921

Failed tests:

test_tools

Failed subtests:

test_freeze_simple_script - test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script
test_pathnames - test.test_tarfile.GzipWriteTest.test_pathnames

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

444 tests OK.

10 slowest tests:

test_tools: 2 min 32 sec
test_concurrent_futures: 2 min 15 sec
test_multiprocessing_spawn: 2 min 12 sec
test_multiprocessing_forkserver: 1 min 26 sec
test_math: 1 min 18 sec
test_signal: 1 min 12 sec
test_multiprocessing_fork: 1 min 10 sec
test_capi: 49.6 sec
test_socket: 47.3 sec
test_gdb: 43.2 sec

1 test failed:
test_tools

20 tests skipped:
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test_devpoll test_ioctl
test_kqueue test_launcher test_msilib test_nis test_peg_generator
test_perf_profiler test_readline test_startfile test_tix
test_tkinter test_ttk test_winconsoleio test_winreg test_winsound
test_wmi test_zipfile64

2 re-run tests:
test_tarfile test_tools

Total duration: 8 min 26 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/test_tarfile.py", line 1469, in test_pathnames
    self._test_pathname(os.path.join("..", "foo"))
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/test_tarfile.py", line 1409, in _test_pathname
    os_helper.create_empty_file(foo)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/support/os_helper.py", line 538, in create_empty_file
    fd = os.open(filename, os.O_WRONLY | os.O_CREAT | os.O_TRUNC)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device: '/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/build/test_python_1713233æ/test_python_worker_1734541æ/@test_1734541_tmpæ-tardir/foo'


Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/test_python_n6t5sdkl/tmpxtrewzr7/cpython/Lib/sysconfig.py", line 740, in get_config_var
    return get_config_vars().get(name)
           ^^^^^^^^^^^^^^^^^
  File "/tmp/test_python_n6t5sdkl/tmpxtrewzr7/cpython/Lib/sysconfig.py", line 723, in get_config_vars
    _init_config_vars()
  File "/tmp/test_python_n6t5sdkl/tmpxtrewzr7/cpython/Lib/sysconfig.py", line 670, in _init_config_vars
    _init_posix(_CONFIG_VARS)
  File "/tmp/test_python_n6t5sdkl/tmpxtrewzr7/cpython/Lib/sysconfig.py", line 536, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata_d_linux_s390x-linux-gnu'


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/test_tools/test_freeze.py", line 27, in test_freeze_simple_script
    outdir, scriptfile, python = helper.prepare(script, outdir)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Tools/freeze/test/freeze.py", line 184, in prepare
    _run_quiet([MAKE, '-C', builddir, 'install'])
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Tools/freeze/test/freeze.py", line 25, in _run_quiet
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/tmp_be2fs30/python-build', 'install']' returned non-zero exit status 2.


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/test_tools/test_freeze.py", line 27, in test_freeze_simple_script
    outdir, scriptfile, python = helper.prepare(script, outdir)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Tools/freeze/test/freeze.py", line 184, in prepare
    _run_quiet([MAKE, '-C', builddir, 'install'])
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Tools/freeze/test/freeze.py", line 25, in _run_quiet
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/test_python_n6t5sdkl/tmpxtrewzr7/python-build', 'install']' returned non-zero exit status 2.


Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/tmp_be2fs30/cpython/Lib/sysconfig.py", line 740, in get_config_var
    return get_config_vars().get(name)
           ^^^^^^^^^^^^^^^^^
  File "/tmp/tmp_be2fs30/cpython/Lib/sysconfig.py", line 723, in get_config_vars
    _init_config_vars()
  File "/tmp/tmp_be2fs30/cpython/Lib/sysconfig.py", line 670, in _init_config_vars
    _init_posix(_CONFIG_VARS)
  File "/tmp/tmp_be2fs30/cpython/Lib/sysconfig.py", line 536, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata_d_linux_s390x-linux-gnu'

bedevere-bot · 2023-05-17T16:34:51Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x has failed when building commit c649df6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/506/builds/4791) and take a look at the build logs.
Check if the failure is related to this commit (c649df6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/506/builds/4791

Failed tests:

test__xxsubinterpreters

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 9, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  11% (1/9)        
remote: Compressing objects:  22% (2/9)        
remote: Compressing objects:  33% (3/9)        
remote: Compressing objects:  44% (4/9)        
remote: Compressing objects:  55% (5/9)        
remote: Compressing objects:  66% (6/9)        
remote: Compressing objects:  77% (7/9)        
remote: Compressing objects:  88% (8/9)        
remote: Compressing objects: 100% (9/9)        
remote: Compressing objects: 100% (9/9), done.        
remote: Total 9 (delta 0), reused 1 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'c649df63e0d052044a4660101d5769ff46ae9234'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at c649df63e0 gh-104372: Cleanup _posixsubprocess `make_inheritable` for async signal safety and no GIL requirement (#104518)
Switched to and reset branch 'main'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Fatal Python error: drop_gil: PyMUTEX_LOCK(gil->switch_mutex) failed
Python runtime state: initialized

Thread 0x00007fa574af5b80 (most recent call first):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 76 in clean_up_interpreters
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 88 in tearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 594 in _callTearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 637 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 690 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 240 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 88 in _run_code
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 198 in _run_module_as_main
Fatal Python error: Aborted

Thread 0x00007fa574af5b80 (most recent call first):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 76 in clean_up_interpreters
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 88 in tearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 594 in _callTearDown
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 637 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 690 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 240 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 88 in _run_code
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters (total: 2)
Aborted (core dumped)
make: *** [Makefile:1977: buildbottest] Error 134

Cannot open file '/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/test-results.xml' for upload

…c signal safety and no GIL requirement (python#104518) Move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process. Much of this was long overdue. We shouldn't have been using PyTuple and PyLong APIs within all of these low level functions anyways.

…or async signal safety and no GIL requirement (pythonGH-104518) Move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process. Much of this was long overdue. We shouldn't have been using PyTuple and PyLong APIs within all of these low level functions anyways.. (cherry picked from commit c649df6) Co-authored-by: Gregory P. Smith <greg@krypto.org>

…c signal safety and no GIL requirement (python#104518) Move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process. Much of this was long overdue. We shouldn't have been using PyTuple and PyLong APIs within all of these low level functions anyways. (cherry picked from commit c649df6)

…or async signal safety and no GIL requirement (pythonGH-104518) Move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process. Much of this was long overdue. We shouldn't have been using PyTuple and PyLong APIs within all of these low level functions anyways.. (cherry picked from commit c649df6) Co-authored-by: Gregory P. Smith <greg@krypto.org>

… signal safety gh-104518 (#104785) Move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process. Much of this was long overdue. We shouldn't have been using PyTuple and PyLong APIs within all of these low level functions anyways. This is a backport of c649df6 for #104518 and the tiny adjustment in d1732fe #104697. Backporting this allows backporting of the real bug fix that requires it. Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>

pythongh-104372: Cleanup make_inheritable for async signal safety.

adae5af

We move all of the Python C API calls into the parent process up front instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from the post-fork/vfork child process.

gpshead self-assigned this May 16, 2023

bedevere-bot added the awaiting core review label May 16, 2023

bedevere-bot mentioned this pull request May 16, 2023

subprocess can block all Python threads when using vfork() until the child process exec() succeeds or fails. #104372

Closed

gpshead added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section labels May 16, 2023

bedevere-bot removed 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section 🔨 test-with-buildbots Test PR w/ buildbots; report in status section labels May 16, 2023

Merge branch 'main' into subprocess-child_exec-cleanup

bfc9d9e

gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 16, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 16, 2023

gpshead added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section labels May 16, 2023

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 16, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 16, 2023

Merge branch 'main' into subprocess-child_exec-cleanup

e073dde

📜🤖 Added by blurb_it.

cd3108c

gpshead merged commit c649df6 into python:main May 17, 2023

bedevere-bot removed the awaiting core review label May 17, 2023

gpshead added the needs backport to 3.11 only security fixes label May 17, 2023

gpshead deleted the subprocess-child_exec-cleanup branch May 18, 2023 03:35

gpshead mentioned this pull request May 20, 2023

gh-104372: Use non-Raw malloc for c_fds_to_keep in _posixsubprocess #104697

Merged

gpshead removed the needs backport to 3.11 only security fixes label May 20, 2023

This was referenced May 23, 2023

[3.11] gh-104372: Cleanup _posixsubprocess make_inheritable for async signal safety gh-104518 #104785

Merged

gh-104372: Drop the GIL around the vfork() call. #104782

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-104372: Cleanup _posixsubprocess `make_inheritable` for async signal safety #104518

gh-104372: Cleanup _posixsubprocess `make_inheritable` for async signal safety #104518

gpshead commented May 16, 2023 •

edited by bedevere-bot

Loading

gpshead commented May 16, 2023

bedevere-bot commented May 16, 2023

bedevere-bot commented May 16, 2023

Eclips4 commented May 16, 2023 •

edited

Loading

bedevere-bot commented May 16, 2023

bedevere-bot commented May 16, 2023

bedevere-bot commented May 16, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

miss-islington commented May 17, 2023

miss-islington commented May 17, 2023

bedevere-bot commented May 17, 2023

bedevere-bot commented May 17, 2023

bedevere-bot commented May 17, 2023

gh-104372: Cleanup _posixsubprocess make_inheritable for async signal safety #104518

gh-104372: Cleanup _posixsubprocess make_inheritable for async signal safety #104518

Conversation

gpshead commented May 16, 2023 • edited by bedevere-bot Loading

gpshead commented May 16, 2023

bedevere-bot commented May 16, 2023

bedevere-bot commented May 16, 2023

Eclips4 commented May 16, 2023 • edited Loading

bedevere-bot commented May 16, 2023

bedevere-bot commented May 16, 2023

bedevere-bot commented May 16, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

gpshead commented May 17, 2023

bedevere-bot commented May 17, 2023

miss-islington commented May 17, 2023

miss-islington commented May 17, 2023

bedevere-bot commented May 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 17, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gh-104372: Cleanup _posixsubprocess `make_inheritable` for async signal safety #104518

gh-104372: Cleanup _posixsubprocess `make_inheritable` for async signal safety #104518

gpshead commented May 16, 2023 •

edited by bedevere-bot

Loading

Eclips4 commented May 16, 2023 •

edited

Loading