test_pdb crashes when run after test_sys_settrace

[serhiy-storchaka]

Crash report

Just run two tests sequentially:

$ ./python -m test -v test_sys_settrace test_pdb
...
test_pdb_breakpoints_preserved_across_interactive_sessions (test.test_pdb)
Doctest: test.test_pdb.test_pdb_breakpoints_preserved_across_interactive_sessions ... python: Python/instrumentation.c:1293: _Py_call_instrumentation_instruction: Assertion `next_opcode != 0' failed.
Fatal Python error: Aborted

Current thread 0x00007fba31132740 (most recent call first):
  File "<string>", line 1 in <module>
  File "/home/serhiy/py/cpython/Lib/bdb.py", line 600 in run
  File "/home/serhiy/py/cpython/Lib/pdb.py", line 1962 in run
  File "<doctest test.test_pdb.test_pdb_breakpoints_preserved_across_interactive_sessions[3]>", line 7 in <module>
  File "/home/serhiy/py/cpython/Lib/doctest.py", line 1374 in __run
  File "/home/serhiy/py/cpython/Lib/doctest.py", line 1524 in run
  File "/home/serhiy/py/cpython/Lib/doctest.py", line 2244 in runTest
  File "/home/serhiy/py/cpython/Lib/unittest/case.py", line 589 in _callTestMethod
  File "/home/serhiy/py/cpython/Lib/unittest/case.py", line 634 in run
  File "/home/serhiy/py/cpython/Lib/unittest/case.py", line 690 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/serhiy/py/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/serhiy/py/cpython/Lib/unittest/runner.py", line 240 in run
  File "/home/serhiy/py/cpython/Lib/test/support/__init__.py", line 1142 in _run_suite
  File "/home/serhiy/py/cpython/Lib/test/support/__init__.py", line 1269 in run_unittest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 393 in run_unittest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 447 in test_func
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 407 in regrtest_runner
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 451 in _load_run_test
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 491 in _runtest_env_changed_exc
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 348 in _runtest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/runtest.py", line 375 in runtest
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 462 in run_test
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 495 in run_tests_sequentially
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 632 in run_tests
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 855 in action_run_tests
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 883 in _main
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 822 in main
  File "/home/serhiy/py/cpython/Lib/test/libregrtest/main.py", line 891 in main
  File "/home/serhiy/py/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/home/serhiy/py/cpython/Lib/runpy.py", line 88 in _run_code
  File "/home/serhiy/py/cpython/Lib/runpy.py", line 198 in _run_module_as_main

It is only reproducible in 3.12+.

Linked PRs

• gh-108976: Make sure instrumentation line returns the correct opcode when instruction instrumentation is stripped #109043
• GH-108976. Keep monitoring data structures valid during de-optimization during callback. #109131
• [3.12] GH-108976. Keep monitoring data structures valid during de-optimization during callback. (GH-109131) #109268

[serhiy-storchaka]

test_pdb only crashes after RaisingTraceFuncTestCase.test_line_event_raises_before_opcode_event or after any of TraceOpcodesTestCase cases from test_sys_settrace.

$ ./python -m test -v test_sys_settrace test_pdb -m test_pdb_display_command -m test.test_sys_settrace.RaisingTraceFuncTestCase.test_line_event_raises_before_opcode_event
$ ./python -m test -v test_sys_settrace test_pdb -m test_pdb_display_command -m test.test_sys_settrace.TraceOpcodesTestCase.test_01_basic

These tests also occur in #108982, so this all may be symptoms of the same bug.

[gaogaotiantian]

I'm able to reduce the MRE:

import sys

sys._getframe().f_trace_opcodes = True

breakpoint()
a = 1

It will bring up the pdb console, and you can just c to trigger the SegFault.

[gaogaotiantian]

The crash is a bit tricky, and affects more than sys.settrace - sys.monitoring also suffers from it.

The fundamental reason was when INSTRUCTION and LINE events are both set, and INSTRUMENTATION is turned off in LINE event callback, the system got confused. In INSTRUMENTED_LINE, it will try to get the next opcode and would be fed a wrong one - INSTRUMENTED_INSTRUCTION by the current design. However, the instrumentation is already stripped, which would eventually cause issues that lead to a SegFault.

I have a fix which simply checks if this case happens when opcode is requested from INSTRUMENTED_LINE and use the original opcode for the instruction.

This might not be the best way to fix it, but it's clean and simple. If @markshannon has a better idea, I can do something different.

[markshannon]

This is fixed now, both in main and 3.12