Inconsistent behavior of asyncio.Server.wait_closed in Python 3.12 versus earlier releases

[tovrstra]

Bug report

Bug description:

The issue fixed in #111424 is only available in Python 3.12. As a result, the example below prematurely closes a connection when using an older Python version. The old behavior may lead to bugs that are hard to catch, and the difference in behavior implies that applications must use workarounds to support multiple Python versions.

If backporting the fix is not an option, it would be helpful to add a note to the documentation of asyncio.Server.wait_closed that it changed in Python 3.12.

Related issues: #104344 (same cause, going from 3.11 to 3.12), #79033

minimal example

server.py

import asyncio
from functools import partial

async def handler(stop_event, reader, writer):
    print("Client connected")
    msg = (await reader.readline()).decode()
    print(f"Received: {msg.strip()}")
    await asyncio.sleep(1)
    if "STOP" in msg:
        print("Client requests stop. Setting stop event")
        stop_event.set()
        await asyncio.sleep(1)
    print("----> Respond with same message <----")
    writer.write(msg.encode())
    await writer.drain()
    writer.close()
    await writer.wait_closed()
    print("----> Closing connection <----")


async def main():
    print("Starting server")
    stop_event = asyncio.Event()
    socket_path = "socket"
    server = await asyncio.start_unix_server(partial(handler, stop_event), socket_path)
    print(f"Server listening at: {socket_path}")
    async with server:
        await stop_event.wait()
        print("Server stopping")
    print("Server stopped")


if __name__ == "__main__":
    asyncio.run(main())

client.py

import asyncio


async def talk(msg):
    socket_path = "socket"
    print(f"Connecting to: {socket_path}")
    reader, writer = await asyncio.open_unix_connection(socket_path)
    print("Connected")
    print(f"Sending message: {msg}")
    writer.write(f"{msg}\n".encode())
    await writer.drain()
    res = (await reader.readline()).decode()
    print(f"Received response: {res[:-1]}")
    print("Closing connection")
    writer.close()
    await writer.wait_closed()


async def main():
    print("Starting client")
    await talk("Hello")
    await talk("STOP")
    print("Stopping client")


if __name__ == "__main__":
    asyncio.run(main())

This is a minimal working example distilled from a more complex use case where the stop_event is set in a function that receives the request and creates the response.

One possible workaround is to postpone setting the stop_event to the end of the handler function. That would be easy in this example, but more involved in the case where I ran into this issue.

Python 3.12 and 3.13 output

server.py

Starting server
Server listening at: socket
Client connected
Received: Hello
----> Respond with same message <----
----> Closing connection <----
Client connected
Received: STOP
Client requests stop. Setting stop event
Server stopping
----> Respond with same message <----
----> Closing connection <----
Server stopped

client.py

Starting client
Connecting to: socket
Connected
Sending message: Hello
Received response: Hello
Closing connection
Connecting to: socket
Connected
Sending message: STOP
Received response: STOP
Closing connection
Stopping client

Python 3.9, 3.10 and 3.11 output

Note that the handler does not complete when it receives STOP.

server.py

Starting server
Server listening at: socket
Client connected
Received: Hello
----> Respond with same message <----
----> Closing connection <----
Client connected
Received: STOP
Client requests stop. Setting stop event
Server stopping
Server stopped

client.py

Starting client
Connecting to: socket
Connected
Sending message: Hello
Received response: Hello
Closing connection
Connecting to: socket
Connected
Sending message: STOP
Received response: 
Closing connection
Stopping client

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13

Operating systems tested on:

Linux

[tovrstra]

Here is a workaround that mimics the implementation in Python 3.12.1 when using a Python version without the fix in #111424. One can add the following code right before print("Server stopped") in server.py:

    if sys.version_info < (3, 12, 1) and server._waiters is not None:
        # Workaround for server.wait_closed() issue fixed in Python 3.12.1
        waiter = server.get_loop().create_future()
        server._waiters.append(waiter)
        await waiter

This may be a useful snippet for implementing a server that should work across different Python versions.