Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove process-id checks from asyncio. Asyncio and fork() does not mix. #2911

Merged
merged 8 commits into from
Sep 14, 2023

Conversation

kristjanvalur
Copy link
Contributor

@kristjanvalur kristjanvalur commented Aug 24, 2023

Pull Request check-list

  • Does $ tox pass with this change (including linting)?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?
  • Was the change added to CHANGES file?

Description of change

Remove PID checks from asyncio connections, since asyncio and fork() does not mix. See issue #2910

Details:

Python does not support fork() on a process with a running asyncio event loop. Therefore supporting a connection pool of connection from a different process is not necessary.

Similarly, connection pools don't need to deal with different threads either.

We remove the checks for process ids, intended to ignore inherited file descriptors (even though these should probably have been closed on fork anyway).

We remove lenience when returning a connection to a pool, it should always be returned to the pool it originated from. Not doing that would be a resource leak.

We simplify BlockingConnectionPool since it does not need to do any multi-threading. It now just adds a simple wait mechanism on top of the ConnectionPool, reducing code and overhead. The QueueClass argument is now ignored. It always was a weird implementation leak anyway

@codecov-commenter
Copy link

codecov-commenter commented Aug 24, 2023

Codecov Report

Patch coverage: 60.52% and project coverage change: -24.53% ⚠️

Comparison is base (1596ac6) 91.28% compared to head (d78b85e) 66.75%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #2911       +/-   ##
===========================================
- Coverage   91.28%   66.75%   -24.53%     
===========================================
  Files         126      126               
  Lines       32398    32331       -67     
===========================================
- Hits        29573    21584     -7989     
- Misses       2825    10747     +7922     
Files Changed Coverage Δ
tests/test_asyncio/conftest.py 49.30% <7.14%> (-35.59%) ⬇️
tests/conftest.py 72.89% <20.00%> (-13.14%) ⬇️
tests/test_asyncio/test_connection_pool.py 66.60% <50.00%> (-25.55%) ⬇️
redis/asyncio/connection.py 79.41% <81.81%> (-3.41%) ⬇️
tests/test_asyncio/test_cluster.py 96.93% <100.00%> (ø)
tests/test_asyncio/test_cwe_404.py 71.42% <100.00%> (-28.58%) ⬇️

... and 76 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kristjanvalur kristjanvalur marked this pull request as ready for review August 24, 2023 13:29
@kristjanvalur kristjanvalur force-pushed the kristjan/async-fork branch 2 times, most recently from d169b49 to 6793fb3 Compare August 29, 2023 13:20
@kristjanvalur kristjanvalur marked this pull request as draft August 29, 2023 13:34
@kristjanvalur kristjanvalur marked this pull request as ready for review August 29, 2023 15:18
@chayim
Copy link
Contributor

chayim commented Sep 11, 2023

@kristjanvalur Just checking in - is this one something you wanted reviewed?

@dvora-h see the test failures. I'm surprised that some of these (cluster) ran.

@chayim chayim added the maintenance Maintenance (CI, Releases, etc) label Sep 11, 2023
@kristjanvalur
Copy link
Contributor Author

kristjanvalur commented Sep 11, 2023

This is not reproducible here with individual steps.
It seems to be some fallout from unittests where the connection pool gets polluted with Mock Connections. One of the changes I did was to not allow returning unknown connections to the pool, since that is a programming error / resource leak. But the unit-tests seem to have been relying on that behaviour.

I'll figure it out and let you know when I think I have something which needs can be reviewed

@kristjanvalur
Copy link
Contributor Author

kristjanvalur commented Sep 12, 2023

@chayim Found the issue. Some mocks were not being undone properly which played havoc with the cleanup. Last two commits are test fixes can be applied to main independently, I can create a separate PR for that if you like.
Please review when you have the time.

@dvora-h dvora-h merged commit ba186d2 into redis:master Sep 14, 2023
53 of 54 checks passed
@kristjanvalur kristjanvalur deleted the kristjan/async-fork branch September 14, 2023 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Maintenance (CI, Releases, etc)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants