Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-41273: asyncio's proactor read transport's better performance by using recv_into instead of recv #21442

Merged
merged 2 commits into from
Jul 14, 2020

Conversation

tontinton
Copy link
Contributor

@tontinton tontinton commented Jul 11, 2020

https://bugs.python.org/issue41273

I got about 150% better performance on await reader.read() using this branch (Windows only).

The way I tested was writing a server / client:
server.py:

import asyncio
import contextlib
import datetime


async def client_connected(reader, writer):
    start = datetime.datetime.now()

    with contextlib.closing(writer):
        for i in range(1000):
            await reader.read(1024 * 64)  # tweak this parameter as much as you like

    diff = datetime.datetime.now() - start
    print(f'{diff.seconds}.{diff.microseconds}')


async def main():
    server = await asyncio.start_server(client_connected, '127.0.0.1', 8888)

    addr = server.sockets[0].getsockname()
    print(f'Serving on {addr}')

    async with server:
        await server.serve_forever()


if __name__ == "__main__":
    asyncio.run(main())

client.py:

import asyncio
import contextlib


async def flood(ip, port):
    message = b'A' * 1024 * 4  # tweak this parameter as much as you like
    reader, writer = await asyncio.open_connection(ip, port)
    with contextlib.closing(writer):
        while True:
            writer.write(message)
            await writer.drain()


if __name__ == "__main__":
    asyncio.run(flood('127.0.0.1', 8888))

By using recv_into instead of recv we do not allocate a new buffer each
time _loop_reading calls recv.

This betters performance for any stream using proactor (basically any
asyncio stream on windows).
By doubling the read buffer size we get better performance.
@1st1 1st1 merged commit 568fb0f into python:master Jul 14, 2020
@bedevere-bot
Copy link

@1st1: Please replace # with GH- in the commit message next time. Thanks!

@1st1
Copy link
Member

1st1 commented Jul 14, 2020

Thank you!

arun-mani-j pushed a commit to arun-mani-j/cpython that referenced this pull request Jul 21, 2020
…using recv_into instead of recv (python#21442)

* bpo-41273: Proactor transport read loop to use recv_into

By using recv_into instead of recv we do not allocate a new buffer each
time _loop_reading calls recv.

This betters performance for any stream using proactor (basically any
asyncio stream on windows).

* bpo-41273: Double proactor read transport buffer size

By doubling the read buffer size we get better performance.
shihai1991 pushed a commit to shihai1991/cpython that referenced this pull request Aug 4, 2020
…using recv_into instead of recv (python#21442)

* bpo-41273: Proactor transport read loop to use recv_into

By using recv_into instead of recv we do not allocate a new buffer each
time _loop_reading calls recv.

This betters performance for any stream using proactor (basically any
asyncio stream on windows).

* bpo-41273: Double proactor read transport buffer size

By doubling the read buffer size we get better performance.
shihai1991 pushed a commit to shihai1991/cpython that referenced this pull request Aug 20, 2020
…using recv_into instead of recv (python#21442)

* bpo-41273: Proactor transport read loop to use recv_into

By using recv_into instead of recv we do not allocate a new buffer each
time _loop_reading calls recv.

This betters performance for any stream using proactor (basically any
asyncio stream on windows).

* bpo-41273: Double proactor read transport buffer size

By doubling the read buffer size we get better performance.
xzy3 pushed a commit to xzy3/cpython that referenced this pull request Oct 18, 2020
…using recv_into instead of recv (python#21442)

* bpo-41273: Proactor transport read loop to use recv_into

By using recv_into instead of recv we do not allocate a new buffer each
time _loop_reading calls recv.

This betters performance for any stream using proactor (basically any
asyncio stream on windows).

* bpo-41273: Double proactor read transport buffer size

By doubling the read buffer size we get better performance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants