Performance overhead of using anyio

[Traktormaster]

Hi!

Preface

Firstly, I'd like to clarify that I've been developing asyncio applications and have not used trio/curio. I have not found the need to really consider using them so far, but different folks prefer one over the other for whatever reason, and that's fine. As far as I understand anyio can be used to make them work together in some capacity in the same program if two different dependencies use one or the other for example. For this reason some libraries have started to just migrate to anyio that have previously been asyncio-only.

Problem

My problem with libraries depending on anyio is that the performance overhead of anyio can be significant while there is absolutely zero benefit for it, because only asncyio is ever utilized in my setups.

I've looked over the libraries that migrated over to anyio and anyio itself. I've quickly concluded that there are many great utilities that are available in anyio, so asking for an anyio-free version (usually by revert) is not even worth trying. After they've migrated to anyio, many custom utilities were dropped that would need to be re-introduced to the tree with tests, that most won't even consider, because anyio provides them. There is no way to convince said libraries to remove anyio for asyncio-only environments.

Measurements

My main gripe is with the get_asynclib private function. It selects the "current" asynchronous backend that is in use, so it is necessarily used all the time. The following code focuses on the difference of using native asyncio code and this backend-selecing wrapper:

import asyncio

import anyio


async def _busy_sleep_asyncio(cl):
    while True:
        await asyncio.sleep(0)
        cl[0] += 1


async def _busy_sleep_anyio(cl):
    while True:
        await anyio.sleep(0)
        cl[0] += 1


async def _check(fn, *args, duration=1):
    cl = [0]
    try:
        await asyncio.wait_for(asyncio.create_task(fn(*args, cl)), timeout=duration)
    except asyncio.TimeoutError:
        return cl[0]
    else:
        raise RuntimeError()


if __name__ == "__main__":
    direct = asyncio.run(_check(_busy_sleep_asyncio))
    print("asyncio:", direct)
    wrapped = asyncio.run(_check(_busy_sleep_anyio))
    print("anyio  :", wrapped)
    print("relative performance with anyio:", round(wrapped * 100 / direct, 2), "%")

The snippet above measures how many times the backend's sleep can be invoked with the native asyncio and the anyio-wrapped asyncio backend. On my machine the output is:

asyncio: 353484
anyio  : 250246
relative performance with anyio: 70.79 %

So compared to natively calling functions there can be an up to 29% performance penalty.

Discussion

I'd like to initiate a discussion about how anyio selects the backend and how it could be improved.

1. Firstly I'd like to ask what a real use-case for selecting the backend dynamically looks like? I'd imagine there are two libraries in the same application and each use a different backend. After starting their own event loops, they can interact provided one or both utilize anyio?
2. If the selected backend could be frozen at the first call, the performance penalty would be more than halved. However, this may not be possible depending on what the answer for 1. might be.
3. I do not want to disturb anyone who uses anyio with my goals here necessarily. I'd also be interested in some global config override (by an environment variable maybe) that selects some alternate/optimized codes for asyncio-only behavior. It would be best that the selected backend is a global object that can be referenced directly, like the asyncio module itself.

[agronholm]

Firstly I'd like to ask what a real use-case for selecting the backend dynamically looks like? I'd imagine there are two libraries in the same application and each use a different backend. After starting their own event loops, they can interact provided one or both utilize anyio?

The most common use case is a trio based app that needs to use a third party asyncio-only library. Thus, some tasks might use asyncio and some might be running trio.

I've been contemplating on providing a method of permanently selecting a back-end via an environment variable, at the obvious cost of compatibility when an app uses two back-ends, for whatever reason. On import, this would replace all the functions with their back-end specific versions, without going through get_asynclib(). Would this suffice?

My long term plan is to get rid of trio altogether by adopting its best parts in asyncio, but that will take a lot of effort and the will to push potentially breaking changes through to the standard library.

[smurfix]

I've been contemplating on providing a method of permanently selecting a back-end via an environment variable

I'd prefer something like

import anyio
anyio.patch_for("trio")

(or anyio.patch_for_trio()), at the top of the main program / module / whatever. But aside from that this is a great idea.

My long term plan is to get rid of trio altogether by adopting its best parts in asyncio

Good luck. You'll certainly need it.

[smurfix]

Firstly I'd like to ask what a real use-case for selecting the backend dynamically looks like? I'd imagine there are two libraries in the same application and each use a different backend. After starting their own event loops, they can interact provided one or both utilize anyio?

If you want to use asyncio and trio in the same program, you need to use a Trio mainloop and supplement that with trio_asyncio and some judicious use of wrappers and/or compatibility hacks to switch from one mode to the other.

Method of last resort, frankly, as trio_asyncio is not exactly bug-free. It also pre-dates the ability to run Trio in "guest mode".

[agronholm]

(or anyio.patch_for_trio()), at the top of the main program / module / whatever. But aside from that this is a great idea.

This becomes a problem if anything else has imported anyio first.

Good luck. You'll certainly need it.

Don't I know that 😏

[smurfix]

This becomes a problem if anything else has imported anyio first.

Well, that's why I wrote to put it at the top of the main program. There's been nobody yet to import anything from.

Besides, the environment variable needs to come from somewhere. You wouldn't want to require a separate wrapper just because your code uses an asyncio-only library and thus requires that backend; on the other hand, if your code sets that envvar before it imports anything (other than os…) you might as well go ahead and import+patch anyio there.

[davidbrochart]

Well, that's why I wrote to put it at the top of the main program. There's been nobody yet to import anything from.

That's also how nest-asyncio works:

import nest_asyncio
nest_asyncio.apply()

[Traktormaster]

I'm glad to hear there is interest in this! The implementation specific selector function by an env-var would be the quickest and easiest improvement, but it does not eliminate the overhead completely. Freezing the backend in the selector improved the performance from 71% to 92%, this is somewhat similar to what would be expected of a contant-function selected by an env-var:

_froze = None
def get_asynclib(asynclib_name: Optional[str] = None) -> Any:
    global _froze
    if _froze is None:
        if asynclib_name is None:
            asynclib_name = sniffio.current_async_library()

        modulename = "anyio._backends._" + asynclib_name
        try:
            _froze = sys.modules[modulename]
        except KeyError:
            _froze = import_module(modulename)

    return _froze

However, adapting my benchmark code a little can demonstrate that using the specific backend directly has zero overhead:

import anyio._backends._asyncio as any_asyncio
async def _busy_sleep_anyio(cl):
    while True:
        await any_asyncio.sleep(0)
        cl[0] += 1

I would suggest that providing zero-overhead in general would be ideal. This could be possible if there was a new canonical import path for the backend like asynio.backend which could either be a direct import alias to a specific backend, or when the dynamic backend selection is desired could be a proxy-object which selects its __dict__ to serve on attribute access.

[Traktormaster]

Around 5% as I've stated:

import asyncio

import anyio


async def _busy_sleep_asyncio(cl):
    while True:
        await asyncio.sleep(0)
        cl[0] += 1


async def _busy_sleep_anyio(cl):
    while True:
        await anyio.sleep(0)
        cl[0] += 1


async def sleep(delay: float) -> None:
    return await asyncio.sleep(delay)


async def _busy_sleep_wrapped(cl):
    while True:
        await sleep(0)
        cl[0] += 1


async def _check(fn, *args, duration=1):
    cl = [0]
    try:
        await asyncio.wait_for(asyncio.create_task(fn(*args, cl)), timeout=duration)
    except asyncio.TimeoutError:
        return cl[0]
    else:
        raise RuntimeError()


if __name__ == "__main__":
    direct = asyncio.run(_check(_busy_sleep_asyncio))
    print("asyncio:", direct)
    wrapped = asyncio.run(_check(_busy_sleep_anyio))
    print("anyio  :", wrapped)
    direct_wrap = asyncio.run(_check(_busy_sleep_wrapped))
    print("wrapped:", direct_wrap)
    print("relative performance with anyio:", round(wrapped * 100 / direct, 2), "%")
    print("relative performance with direct-wrapper:", round(direct_wrap * 100 / direct, 2), "%")

Output:

asyncio: 349538
anyio  : 248374
wrapped: 334808
relative performance with anyio: 71.06 %
relative performance with direct-wrapper: 95.79 %

[agronholm]

As a first step, 71.06 -> 95.79 % doesn't seem so bad, especially considering how little absolute overhead it has.

[Traktormaster]

Yes, it's a considerable improvement, that I'd be glad to have. A zero-overhead solution could be easier to create later, once this is done.

[agronholm]

Revisiting this, the script gives very inconsistent results. I found this script a bit more reliable:

import asyncio
import time

import anyio


async def main(sleep_fn):
    start = time.monotonic_ns()
    for _ in range(100000):
        await sleep_fn(0)

    return time.monotonic_ns() - start


if __name__ == "__main__":
    asyncio_duration = asyncio.run(main(asyncio.sleep))
    print(f"asyncio: {asyncio_duration}")
    anyio_duration = asyncio.run(main(anyio.sleep))
    print(f"anyio: {anyio_duration}")
    print("anyio's performance overhead:", round((anyio_duration - asyncio_duration) / asyncio_duration * 100, 2), "%")

When run against the current master, I get a median overhead of ~45%.
When implementing "locking" the async backend, the overhead drops to ~17%.
If I skip get_async_backend() entirely and just call asyncio.sleep() directly from anyio._eventloop.sleep(), the overhead drops to ~6%.

I also ran just the anyio portion with strace. Without the async backend locking in place, I got 202574 system calls, while with locking, I got 102574 calls. So locking avoids roughly half of the system calls, at least in this somewhat contrived case.

I then started digging deeper and found that there were a lot of getpid() syscalls, triggered by sniffio.current_async_library(). It turns out that was due to asyncio.get_running_loop() needing to access thread locals.

[agronholm]

If you like, I have #736 for you to evaluate. Let me know if you do try it.

[rafalkrupinski]

Would it be possible to make a light version of anyio, one that merely tries to import trio and falls back to asyncio, and just copies implementations straight from a backend?
Is there any downside to it?

[agronholm]

I don't understand what you're talking about here. Your original comment suggested trying to import trio, and then falling back to asyncio, and that's what I commented on.

[rafalkrupinski]

OK, let's try again...

What do you think of an alternative approach, a per this code

[agronholm]

OK, let's try again...

What do you think of an alternative approach, a per this code

Well, of course it's not always as simple as replacing a free function with something from the async backend, but assuming it was, all we would gain is one less stack frame per call. Can you demonstrate that this makes enough of a real world difference that it's worth the effort?

[rafalkrupinski]

Oh, now I see how simple your change is.👍

[rafalkrupinski]

It's opt-in until it breaks.

Consider this setup (there are others).

Your program uses anyio, and you install the Trio backend. Your program also uses the anyio-supporting library A.

Library A needs to call B which is asyncio-only, so it uses sniffio to discover that it's on Trio, then wraps the calls to B with trio-asyncio.

B happens to have been ported to anyio in its latest release, no deprecations, so they didn't increase the major version number.

You update B.

Boom.

@smurfix
These are interesting problems you've brought up

The first issue is an obvious bug in A - an anyio library that might be running under trio is using asyncio code. To me it's the same kind of a problem as a library trying to read a non-existing file and raising unexpected FileNotFoundError.

The second issue is a problem with B - a significant change in behaviour without increasing the major number.

I think it's alright to protect users from such bugs if it's either cheap or it's opt-in.

[smurfix]

On 07.07.24 23:08, Raphael Krupinski wrote: When using |trio-asyncio| anyio should run with either backend, no?

It doesn't know whether there's a Trio runner in control of the asyncio "loop", and testing for that situation would add even more overhead.

…

-- -- mit freundlichen Grüßen -- -- Matthias Urlichs

[rafalkrupinski]

On 07.07.24 23:08, Raphael Krupinski wrote: When using |trio-asyncio| anyio should run with either backend, no?
It doesn't know whether there's a Trio runner in control of the asyncio "loop", and testing for that situation would add even more overhead.

I made that comment before I knew anything about trio-asyncio, based only on the header in its documentation:
trio-asyncio: A re-implementation of the asyncio mainloop on top of Trio

Of course, if you actually have two event loops and bridge them with trio-asyncio, locking wouldn't work

[jruere]

Hi! I'm new to this discussion, so apologies in advance for my likely confusion.

I understand that while using trio, asyncio could also be indirectly in use, so we cannot avoid the per-call backend dispatch.

Is this also possible when using asyncio? Could trio be also in use?

[smurfix]

In a different thread, or with Trio guest mode, yes.

AFAIK nobody has written a trio-on-asyncio guest mode module yet, but there's no reason why that can't be done.

[jruere]

The price of being a super dynamic language, I guess.

How about expecting a single backend but allowing the user to configure AnyIO to remain dynamic? This could be sensible if 90% of the users would benefit and the users of these mixed backends would not be unexpected (e.g. a module just does this, and you never knew).

Alternatively, caching results based on something cheap to calculate. Is there a Task ID? Something like that?
If a Task were always run in the event loop where it was created, then perhaps the Task could have a reference to its Backend?

[agronholm]

In my own tests, only pathological, unrealistic benchmarks show any notable performance improvements when sniffio is skipped, so I'm not 100% sure this is something we want to focus on. AnyIO does have performance issues elsewhere. I just tested the performance of anyio.Lock and it's much slower than the respective native implementations.

[jruere]

The PRs referenced from #760 have a benchmark of HTTPX, which is greatly affected by synchronization primitives.

[agronholm]

Here I have a PR that pretty much eliminates the locking overhead of AnyIO vs asyncio: #761