-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Async Support #1899
Add Async Support #1899
Conversation
* We need Python 3.7 from async-timeout plus 3.6 EOLed * test_commands not implemented yet * Added uvloop support (hopefully)
Codecov Report
@@ Coverage Diff @@
## master #1899 +/- ##
==========================================
- Coverage 92.94% 92.69% -0.25%
==========================================
Files 78 99 +21
Lines 16785 20717 +3932
==========================================
+ Hits 15600 19204 +3604
- Misses 1185 1513 +328
Continue to review full report at Codecov.
|
@chayim I think for the most part, everything from aioredis has been moved. I type annotated commands. Just need to resolve Redis Modules, cluster support, and CI errors (you can take a look at my aioredis PR unless I forgot to push my changes; in which case, the fashion is similar with Protocols, similar to swift protocols). Besides that my midterms are coming up so won't have time to resolve this mess for a bit. Please ping me in this PR or other comm channels if you've got Qs! There's also no need to de-duplicate all the code in this PR. The primary goal of this pr in my opinion is to just add support first. Then we can nitpick |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @chayim
We'd appreciate if this can be merged in redis-py so that async APIs can be supported too. It will be a great addition.
@pssolanki111 Yup, this is the plan. Between COVID and thing, it's slowed a bit - but the goal is to merge this in. Just want to ensure all tests currently run in the harness (they don't). @dvora-h is working on that. Thanks for the confirmation! |
@pssolanki111 @Andrew-Chen-Wang can you folks help us out? In GH actions (and locally on my Linux desktops) we run into the following event loop issue, across python versions. You're marked the tests with pytest.mark.xfail which I would assume means they can fail, and life can go on safely. At the same time, they're not. Do you have any thoughts - I'm in no way an async expert and will only start to dig in.
|
@chayim I think multiprocessing is the issue inherently. Passing the connnection object probably passed the loop from the master process. The practice of passing the object isn’t stable and I think it should be removed.
…________________________________
From: Chayim ***@***.***>
Sent: Wednesday, February 9, 2022 8:58:33 AM
To: redis/redis-py ***@***.***>
Cc: Andrew Chen Wang ***@***.***>; Mention ***@***.***>
Subject: Re: [redis/redis-py] Add Async Support (PR #1899)
@pssolanki111<https://github.com/pssolanki111> @Andrew-Chen-Wang<https://github.com/Andrew-Chen-Wang> can you folks help us out? In GH actions (and locally on my Linux desktops) we run into the following event loop issue, across python versions. You're marked the tests with pytest.mark.xfail which I would assume means they can fail, and life can go on safely. At the same time, they're not. Do you have any thoughts - I'm in no way an async expert and will only start to dig in.
tests/test_asyncio/test_multiprocessing.py Process Process-10:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/data/repos/redis/redis-py/asyncio/tests/test_asyncio/test_multiprocessing.py", line 51, in target
asyncio.get_event_loop().run_until_complete(atarget(conn))
File "/usr/lib/python3.10/asyncio/base_events.py", line 617, in run_until_complete
self._check_running()
File "/usr/lib/python3.10/asyncio/base_events.py", line 577, in _check_running
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
xProcess Process-11:
—
Reply to this email directly, view it on GitHub<#1899 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOLG4VVNMONL5XPCFGYYSGLU2JXITANCNFSM5MWZ4HGA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
@chayim As @Andrew-Chen-Wang pointed out, it's an issue with multiprocessing. More specifically it appears to be a race condition with the event loop. I have not looked at the source, only looking at the source; looks like an attempt to run multiple event loops in multiple processes? If so, that will not work. Processes in python have some veeeeery wierd behaviors (I've been bruised by it a lot, so I know). so the suggestions are,
I'm interested to know though why we even have mp in the tests in the first place. |
@pssolanki111 aioredis is just a port of redis py, so the tests mustve come with it. cc @seandstewart if you'd like to join in on the fun. |
@Andrew-Chen-Wang ahhh that makes sense. That just means there is no need for those tests. Async behavior with mp is NOT recommended, nor it is a good design pattern. Have i used mp or threading with redis-py? yes. will I use mp with aioredis? hell no. |
@pssolanki111 You raise a great point here. multiproc with async really isn't a great design pattern. My limited async experience agrees. Keeping that in mind, pushing up a removal for the multiprocessing async tests, and running through. |
Any chance someone (plural!) would love to help lift the examples into the examples folder? Today, we accomplish this by editing docs/examples.rst, and adding new jupyter notebooks in the docs/examples folder. I think if we do this - we're going to have an amazing PR! In my mind, when this lands - I could easily see a 4.2.0beta1 coming out of this. What does everyone think? |
this might just be me but i'm not certain what exactly you meant there : D |
I'm very excited to see such progress on this PR. I do want to stress that we have quite a few issues which have emerged out of airodes-py's port to align with the implementation here. aio-libs-abandoned/aioredis-py#1225 So after this initial move, there's plenty of room for improvement. One thing I've been digging into is our degraded performance compared to aioredis v1.3 (the current impl is between 4-10x slower on simple get/sets) - there are some major reasons why we moved away from that code from a design and maintenance standpoint, but I think there's a lot we can learn from it about our parsing & IO-bound workloads. |
I think it is a good idea to get this PR merged first. That way, we can quickly add other things from aio-libs-abandoned/aioredis-py#1225 such as aio-libs-abandoned/aioredis-py#1207, aligning the constructors for important classes like Redis and ConnectionPool, type annotations, and cluster support. I also agree with seandstewart. The performance degradation is a major factor for many organizations not to upgrade aioredis. Having any assistance for Sean to resolve the out-of-order-response issue would be great (I will also look into it once I find the time)! Thank you so much for helping keep asyncio for Redis alive! |
We should think about feature-flagging async support to 3.7+ (I'd even argue potentially 3.8+). There were numerous changes and bugfixes to asyncio and concurrent.futures which directly affect our ability to serve a quality asyncio-based client. |
@seandstewart What are you suggesting? Having classes within a single library that is dependent on a specific version of python strikes me as both difficult, and poor form - but perhaps I misunderstood you. Can you educate me? |
fixing async tests core.command typing that broke command tests
On my phone, so quick review: supported_errors in retry.py should be Tuple[Type[Error]]. When grabbing the loop, we can make a fn instead called "get_event_loop" |
Two different sets of issues are left:
|
updating docs to point to async connections adding python deprecation notice
Dear everyone who has been involved.... Everything now passes. Things are more sane! Seriously though, we'll get this in (today)! We'll clean up and merge another couple of PRs, and 4.2.0rc1 is on it's way. A hearty THANK YOU to everyone involved. I'm so grateful for the amazing community we have, and looking forward to our future collaboration! |
In order to improve the compatibility for python 3.11, aioredis is replaced by redis. The package has been deprecated in Feb 21, 2023 since the introduction of aioredis into redis package with the 4.2.0rc1 release. For more info see: - redis/redis-py#1899 - https://github.com/aio-libs/aioredis-py#-aioredis-is-now-in-redis-py-420rc1-
In order to improve the compatibility for python 3.11, aioredis is replaced by redis. The package has been deprecated in Feb 21, 2023 since the introduction of aioredis into redis package with the 4.2.0rc1 release. For more info see: - redis/redis-py#1899 - https://github.com/aio-libs/aioredis-py#-aioredis-is-now-in-redis-py-420rc1-
Adds initial async support from aioredis.
Things to do:
TODO in other PRs: