GUI tests: CPython 3.9 CI failure for macOS: macOS 11 (1107) or later required, have instead 11 (1106) !

[corona10]

CI failure history

• https://github.com/python/cpython/actions/runs/3069822692/jobs/4958905267
• https://github.com/python/cpython/actions/runs/3105434391/jobs/5032588687

CI log

test_write_stdout (idlelib.idle_test.test_squeezer.SqueezerTest)
macOS 11 (1107) or later required, have instead 11 (1106) !
Fatal Python error: Aborted

Current thread 0x000000010df4ee00 (most recent call first):
  File "/Users/runner/work/cpython/cpython/Lib/tkinter/__init__.py", line 2270 in __init__
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/macosx.py", line 23 in _init_tk_type
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/macosx.py", line 42 in isAquaTk
Test Squeezer's overriding of the EditorWindow's write() method. ... 
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/squeezer.py", line 116 in __init__
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/squeezer.py", line 260 in mywrite
  File "/Users/runner/work/cpython/cpython/Lib/idlelib/idle_test/test_squeezer.py", line 192 in test_write_stdout
  File "/Users/runner/work/cpython/cpython/Lib/unittest/case.py", line 550 in _callTestMethod
  File "/Users/runner/work/cpython/cpython/Lib/unittest/case.py", line 592 in run
  File "/Users/runner/work/cpython/cpython/Lib/unittest/case.py", line 651 in __call__
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
  File "/Users/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/runner/work/cpython/cpython/Lib/unittest/runner.py", line 184 in run
  File "/Users/runner/work/cpython/cpython/Lib/test/support/__init__.py", line 1850 in _run_suite
  File "/Users/runner/work/cpython/cpython/Lib/test/support/__init__.py", line 1974 in run_unittest
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 263 in _test_module
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 288 in _runtest_inner2
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 326 in _runtest_inner
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 217 in _runtest
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 247 in runtest
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 334 in rerun_failed_tests
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 716 in _main
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 672 in main
  File "/Users/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 733 in main
  File "/Users/runner/work/cpython/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/Users/runner/work/cpython/cpython/Lib/runpy.py", line 87 in _run_code
  File "/Users/runner/work/cpython/cpython/Lib/runpy.py", line 197 in _run_module_as_main
make: *** [buildbottest] Abort trap: 6

Linked PRs

• [3.10] gh-97032: avoid test_squeezer crash on macOS buildbots #115508
• [3.9] gh-97032: avoid test_squeezer crash on macOS buildbots (GH-115508) #115655
• [3.8] [3.10] gh-97032: avoid test_squeezer crash on macOS buildbots (GH-115508) #115656

[corona10]

cc @ned-deily @ronaldoussoren @vstinner

[zooba]

I'm seeing this for 3.10 on some of my internal builds since the build machines flipped over from 1106 to 1107. So is one of the checks wrong in the test?

[corona10]

So is one of the checks wrong in the test?

The GitHub Action uses 11.07 version:

Operating System
  macOS
  11.7
  20G817

Yeah.. l presume that the checking would be wrong.. but I need more clues.

[ronaldoussoren]

The action seems to use a very old version of Tk (probably the system install):

tkinter.TCL_VERSION: 8.5
tkinter.TK_VERSION: 8.5
tkinter.info_patchlevel: 8.5.9

This is from the "Display Build Info" step of the first linked CI failure.

I'm inclined to blame Tk here given that the crash happens while testing code that uses Tkinter.

Looks like we could update the action to "brew install" a newer version of Tcl/Tk.

[corona10]

@ronaldoussoren

Looks like we could update the action to "brew install" a newer version of Tcl/Tk.

The interesting fact is that the latest tcl-tk is already installed.
The problem might be which version was linked to the configuration.
My patch solves the problem: #97525

This issue only happens at the 3.9 branch,
The difference between 3.9 and 3.10+ is whether a25dcae is applied or not.
The difference between 3.7-3.8 and 3.9 is the version of actions/checkout.
From 3.7 to 3.8 branch use the actions/checkout@2 and the 3.9 branch use the actions/checkout@3

[ned-deily]

The issue here is that how Tcl/Tk libraries are detected changed between 3.9 (and earlier branches) and 3.10, with a further change in 3.11. As documented, 3.10 introduced implicit support for using pkg-config to find Tcl and Tk headers and libraries. Prior to 3.10, the explicit with-tcltk-includes and with-tcltk-libs were used to override default locations, i.e. like for overriding the use of the long-broken and deprecated Tcl/Tk 8.5 frameworks shipped by Apple with macOS. (Note for 3.11, with the extensive work to reduce and eventually remove the use of Distutils in building cpython itself, the option to use the old explicit options have been removed in favor of pkg-config and the new generalized options available for most extension modules. So for 3.9 and earlier, restoring the old explicit options as proposed in #97525 should fix the macOS CI builds.

[ned-deily]

FTR, Tk-related GUI tests are skipped during macOS CI runs on other branches because the Python build is not an installed framework build and the Homebrew-supplied Tcl and Tk libs are not from a Tcl/Tk framework build, so the tests are not run under a GUI-enabling app bundle, unlike when falling back to the legacy and faulty Apple-supplied Tcl/Tk 8.5 frameworks as was happening here.

[terryjreedy]

#97530 patches idlelib.macosx._init_tk_type to no longer call tkinter.__init__.Tk when it will crash. Someone other than me will have to backport it past 3.10 if anyone wishes. If so, that should be done before this issue is fixed. I suggest that this issue include adding a direct test that macOS CI testing is done with GUI available, rather than depending on an accidental IDLE bug for detection. Same for other systems?

[ned-deily]

I suggest that this issue include adding a direct test that macOS CI testing is done with GUI available, rather than depending on an accidental IDLE bug for detection.

This doesn't sound like a security-related issue and surely nothing has changed with regard to IDLE tests in security-fix-only branches so I don't think there is any need to backport #97530. And there are already GUI-available tests in the TK tests, IIRC.

[terryjreedy]

For 3.9, the second link above shows that test_tcl, test_tk, and test_ttk_guionly all passed as 'skipped'. The crash of a non-gui IDLE test due to it very indirectly invoking a Tk() call was AFAIK the only failure indication. So leaving the test unfixed in 3.9- is fine with me.

My main concern is with current branches and they seem to be the same. On my PR, the Mac test of the newly combined test_tkinter passes as skipped.

0:02:37 load avg: 4.44 [ 92/437] test_tkinter skipped (resource denied)
test_tkinter skipped -- cannot run without OS X gui process

I don't know what triggered the failure reported here. If it is something Apple did, then having CI fail until we adjust would be unpleasant. If it was a PR that got committed, then in my opinion the PR should have been flagged.

[terryjreedy]

Ned, 2 above:

I don't think there is any need to backport #97530

Good thing you did not as it is buggy. #97527 (comment)

[ronaldoussoren]

Is there anything left to do for this issue?

If I read the conversation correctly the crash does not happen with versions of Python beyond 3.9 and the change that fixes the crash is not a security fix and won't be back ported to 3.9.

I'd say this means the issue can be closed.

[corona10]

Yeah let’s closing it

[encukou]

Marking as 3.10 release blocker, for details see: #115508 (comment)

[encukou]

Crashing test is now skipped in source-only branches. Not an ideal fix, but an adequate one.
See the PR for discussions.