termios.tcdrain hangs on MacOS

[benthorner]

Bug report

The following Python program hangs forever on MacOS, but does not hang on Linux:

import os, termios, time
device_fd, tty_fd  = os.openpty()

def loop():
    while True:
        data = os.read(device_fd, 1)
        print(data)  # <--- never gets here
        time.sleep(.1)

from threading import Thread
thread = Thread(target=loop, daemon=True)
thread.start()

os.write(tty_fd, b'123')
termios.tcdrain(tty_fd)  # <--- gets stuck here
time.sleep(3)  # allow thread to read all data

Reading the data in a thread should mean there's absolutely no reason for tcdrain to hang. The docs say it only waits for the data to be "transmitted", which sounds different to "read". And there's the fact this works on Linux.

Any single-threaded call to tcdrain() using tty_fd also hangs, even if the os.write line is removed i.e. there's nothing for tcdrain to wait for write. When I try with a real file descriptor or a pipe I get an error: 'Inappropriate ioctl for device' - it's unclear if it's the combination of tcdrain and openpty or just tcdrain that's causing the issue.

When I sample the process using the MacOS Activity Monitor I get the following at the bottom of the call graph for the main thread (unchanging) and the secondary "read" thread (bottommost lines vary per sample) respectively:

...
+ 2553 termios_tcdrain  (in termios.cpython-39-darwin.so) + 56  [0x1048dedcc]
+   2553 tcdrain  (in libsystem_c.dylib) + 48  [0x19f27c454]
+     2553 ioctl  (in libsystem_kernel.dylib) + 36  [0x19f30b0c0]
+       2553 __ioctl  (in libsystem_kernel.dylib) + 8  [0x19f30b0d4]
...
  853 os_read  (in python3.9) + 320  [0x100d45ad8]
    853 _Py_read  (in python3.9) + 92  [0x100d35ca0]
      853 PyEval_RestoreThread  (in python3.9) + 24  [0x100cd2c7c]
        853 take_gil  (in python3.9) + 176  [0x100cd2550]
          852 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1236  [0x19f34483c]
            ! 852 __psynch_cvwait  (in libsystem_kernel.dylib) + 8  [0x19f30a270]
              1 _pthread_cond_wait  (in libsystem_pthread.dylib) + 344  [0x19f3444c0]
                1 __gettimeofday  (in libsystem_kernel.dylib) + 12  [0x19f30aa0c]

The questions I'm struggling to answer:

• What is the expected behaviour of tcdrain in this scenario?
• If the Linux behaviour is expected, why does it not work on MacOS?

Your environment

• CPython versions tested on: 3.9.13
• Operating system and architecture: MacOS 12.5.1 (ARM CPU).

I've also tested on MacOS with an Intel CPU and another developer has tested on Linux (to prove it terminates there). Originally I thought the issue might be an OS issue and posted in the Apple dev forum about it.

Linked PRs

• GH-97001: Release GIL in termios extension #99503
• [3.11] GH-97001: Release GIL in termios extension (GH-99503) #99679
• [3.10] GH-97001: Release GIL in termios extension (GH-99503) #99680

[ronaldoussoren]

This could be a difference between Linux and macOS, possibly with a different definition of what 'transmitted to the terminal' means. The termios library is a thin wrapper around a system API and inherits its behaviour from the system.

The implementation of termios.tcdrain does not release the GIL, which might be related: main thread is waiting for the tcdrain(3), while the secondary thread doesn't make progress because it cannot acquire the GIL.

I've done an experiment with a patched version of the termios extension that releases the GIL during calls to tcdrain and that fixes the issue for me. I'll see if I can create a PR that releases the GIL for all functions in the termios extension during the weekend (available time being the main challenge). Note that such a bug fix won't be backported to 3.9, that version is in security fix mode and this is not a security issue.

[benthorner]

@ronaldoussoren sounds great! This one has been puzzling me for ages, so I'm really pleased to hear you made some progress with it. Even if it's not fixed for a while, people can still see this if they come across it.

One thing still puzzles me: when should we use tcdrain? When I've removed it in my experiments I get the expected outcome, which indicates it adds no value. Do you have any ideas about scenarios to use it in?

[ronaldoussoren]

I have no idea to be honest, I barely use ptys myself and have never used termios. The Linux manpage suggests that the API waits until all data in the kernel buffers have been send. That's probably more noticeable when using a real TTY device.