Disable keep-alive when working with proxy #3070

asvetlov · 2018-06-11T13:49:12Z

Many HTTP proxies has buggy keepalive support.

Let's not reuse connection but close it after processing every response.

socketpair · 2018-06-11T14:08:08Z

I disagree about that. This is performance issue. Could you provide exact cases where that problem reproduced ?

asvetlov · 2018-06-11T14:16:01Z

The feature was disabled from the very beginning.
I've enabled it in version 3.3 but we have constant timeout issues on our production servers.

Would be nice to have opt-in flag but it requires another issue/PR (maybe an attribute in ProxyInfo dataclass).

socketpair · 2018-06-11T14:19:04Z

Maybe bug in your production servers ? does not it ?

asvetlov · 2018-06-11T14:25:02Z

No.
It is definitely a bug of our proxies (force_close for connector removes all timeout issues).
I recall the same problems 4 years ago (different project and proxy provider).
I suspect that buggy proxies are pretty common.

codecov-io · 2018-06-11T19:11:51Z

Codecov Report

Merging #3070 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3070      +/-   ##
==========================================
+ Coverage   98.11%   98.11%   +<.01%     
==========================================
  Files          42       42              
  Lines        7747     7750       +3     
  Branches     1349     1349              
==========================================
+ Hits         7601     7604       +3     
  Misses         51       51              
  Partials       95       95

Impacted Files	Coverage Δ
aiohttp/client_proto.py	`96% <100%> (+0.05%)`	⬆️
aiohttp/connector.py	`98.18% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3dfc123...3eba4b3. Read the comment docs.

* Disable keep-alive when working with proxy * Add changenote (cherry picked from commit 8b9924f) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

This PR updates [aiohttp](https://pypi.org/project/aiohttp) from **3.3.1** to **3.3.2**. <details> <summary>Changelog</summary> ### 3.3.2 ``` ================== - Many HTTP proxies has buggy keepalive support. Let's not reuse connection but close it after processing every response. (`3070 <https://github.com/aio-libs/aiohttp/pull/3070>`_) - Provide vendor source files in tarball (`3076 <https://github.com/aio-libs/aiohttp/pull/3076>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/aiohttp - Changelog: https://pyup.io/changelogs/aiohttp/ - Repo: https://github.com/aio-libs/aiohttp </details>

================== Features -------- - Add type hints (`3049 <https://github.com/aio-libs/aiohttp/pull/3049>`_) - Add ``raise_for_status`` request parameter (`3073 <https://github.com/aio-libs/aiohttp/pull/3073>`_) - Add type hints to HTTP client (`3092 <https://github.com/aio-libs/aiohttp/pull/3092>`_) - Minor server optimizations (`3095 <https://github.com/aio-libs/aiohttp/pull/3095>`_) - Preserve the cause when `HTTPException` is raised from another exception. (`3096 <https://github.com/aio-libs/aiohttp/pull/3096>`_) - Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 <https://github.com/aio-libs/aiohttp/pull/3104>`_) - Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 <https://github.com/aio-libs/aiohttp/pull/3173>`_) - The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 <https://github.com/aio-libs/aiohttp/pull/3183>`_) Bugfixes -------- - Prevent double closing when client connection is released before the last ``data_received()`` callback. (`3031 <https://github.com/aio-libs/aiohttp/pull/3031>`_) - Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 <https://github.com/aio-libs/aiohttp/pull/3051>`_) - Postpone web task creation to connection establishment. (`3052 <https://github.com/aio-libs/aiohttp/pull/3052>`_) - Fix ``sock_read`` timeout. (`3053 <https://github.com/aio-libs/aiohttp/pull/3053>`_) - When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 <https://github.com/aio-libs/aiohttp/pull/3054>`_) - fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 <https://github.com/aio-libs/aiohttp/pull/3062>`_) - correct filename in content-disposition with multipart body (`3064 <https://github.com/aio-libs/aiohttp/pull/3064>`_) - Many HTTP proxies has buggy keepalive support. Let's not reuse connection but close it after processing every response. (`3070 <https://github.com/aio-libs/aiohttp/pull/3070>`_) - raise 413 "Payload Too Large" rather than raising ValueError in request.post() Add helpful debug message to 413 responses (`3087 <https://github.com/aio-libs/aiohttp/pull/3087>`_) - Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 <https://github.com/aio-libs/aiohttp/pull/3100>`_) - Fix server request objects comparison (`3116 <https://github.com/aio-libs/aiohttp/pull/3116>`_) - Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 <https://github.com/aio-libs/aiohttp/pull/3123>`_) - Fix timeout precondition checkers (`3145 <https://github.com/aio-libs/aiohttp/pull/3145>`_) Improved Documentation ---------------------- - Add a new FAQ entry that clarifies that you should not reuse response objects in middleware functions. (`3020 <https://github.com/aio-libs/aiohttp/pull/3020>`_) - Add FAQ section "Why is creating a ClientSession outside of an event loop dangerous?" (`3072 <https://github.com/aio-libs/aiohttp/pull/3072>`_) - Fix link to Rambler (`3115 <https://github.com/aio-libs/aiohttp/pull/3115>`_) - Fix TCPSite documentation on the Server Reference page. (`3146 <https://github.com/aio-libs/aiohttp/pull/3146>`_) - Fix documentation build configuration file for Windows. (`3147 <https://github.com/aio-libs/aiohttp/pull/3147>`_) - Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 <https://github.com/aio-libs/aiohttp/pull/3151>`_) - Mention that ``app.make_handler`` is deprecated, recommend to use runners API instead. (`3157 <https://github.com/aio-libs/aiohttp/pull/3157>`_) Deprecations and Removals ------------------------- - Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 <https://github.com/aio-libs/aiohttp/pull/2826>`_) - Drop ``reader`` parameter from ``request.multipart()``. (`3090 <https://github.com/aio-libs/aiohttp/pull/3090>`_)

This PR updates [aiohttp](https://pypi.org/project/aiohttp) from **3.3.2** to **3.4.4**. <details> <summary>Changelog</summary> ### 3.4.4 ``` ================== - Fix installation from sources when compiling toolkit is not available (`3241 <https://github.com/aio-libs/aiohttp/pull/3241>`_) ``` ### 3.4.3 ``` ================== - Add ``app.pre_frozen`` state to properly handle startup signals in sub-applications. (`3237 <https://github.com/aio-libs/aiohttp/pull/3237>`_) ``` ### 3.4.2 ``` ================== - Fix ``iter_chunks`` type annotation (`3230 <https://github.com/aio-libs/aiohttp/pull/3230>`_) ``` ### 3.4.1 ``` ================== - Fix empty header parsing regression. (`3218 <https://github.com/aio-libs/aiohttp/pull/3218>`_) - Fix BaseRequest.raw_headers doc. (`3215 <https://github.com/aio-libs/aiohttp/pull/3215>`_) - Fix documentation building on ReadTheDocs (`3221 <https://github.com/aio-libs/aiohttp/pull/3221>`_) ``` ### 3.4.0 ``` ================== Features -------- - Add type hints (`3049 <https://github.com/aio-libs/aiohttp/pull/3049>`_) - Add ``raise_for_status`` request parameter (`3073 <https://github.com/aio-libs/aiohttp/pull/3073>`_) - Add type hints to HTTP client (`3092 <https://github.com/aio-libs/aiohttp/pull/3092>`_) - Minor server optimizations (`3095 <https://github.com/aio-libs/aiohttp/pull/3095>`_) - Preserve the cause when `HTTPException` is raised from another exception. (`3096 <https://github.com/aio-libs/aiohttp/pull/3096>`_) - Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 <https://github.com/aio-libs/aiohttp/pull/3104>`_) - Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 <https://github.com/aio-libs/aiohttp/pull/3173>`_) - The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 <https://github.com/aio-libs/aiohttp/pull/3183>`_) Bugfixes -------- - Prevent double closing when client connection is released before the last ``data_received()`` callback. (`3031 <https://github.com/aio-libs/aiohttp/pull/3031>`_) - Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 <https://github.com/aio-libs/aiohttp/pull/3051>`_) - Postpone web task creation to connection establishment. (`3052 <https://github.com/aio-libs/aiohttp/pull/3052>`_) - Fix ``sock_read`` timeout. (`3053 <https://github.com/aio-libs/aiohttp/pull/3053>`_) - When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 <https://github.com/aio-libs/aiohttp/pull/3054>`_) - fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 <https://github.com/aio-libs/aiohttp/pull/3062>`_) - correct filename in content-disposition with multipart body (`3064 <https://github.com/aio-libs/aiohttp/pull/3064>`_) - Many HTTP proxies has buggy keepalive support. Let's not reuse connection but close it after processing every response. (`3070 <https://github.com/aio-libs/aiohttp/pull/3070>`_) - raise 413 "Payload Too Large" rather than raising ValueError in request.post() Add helpful debug message to 413 responses (`3087 <https://github.com/aio-libs/aiohttp/pull/3087>`_) - Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 <https://github.com/aio-libs/aiohttp/pull/3100>`_) - Fix server request objects comparison (`3116 <https://github.com/aio-libs/aiohttp/pull/3116>`_) - Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 <https://github.com/aio-libs/aiohttp/pull/3123>`_) - Fix timeout precondition checkers (`3145 <https://github.com/aio-libs/aiohttp/pull/3145>`_) Improved Documentation ---------------------- - Add a new FAQ entry that clarifies that you should not reuse response objects in middleware functions. (`3020 <https://github.com/aio-libs/aiohttp/pull/3020>`_) - Add FAQ section "Why is creating a ClientSession outside of an event loop dangerous?" (`3072 <https://github.com/aio-libs/aiohttp/pull/3072>`_) - Fix link to Rambler (`3115 <https://github.com/aio-libs/aiohttp/pull/3115>`_) - Fix TCPSite documentation on the Server Reference page. (`3146 <https://github.com/aio-libs/aiohttp/pull/3146>`_) - Fix documentation build configuration file for Windows. (`3147 <https://github.com/aio-libs/aiohttp/pull/3147>`_) - Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 <https://github.com/aio-libs/aiohttp/pull/3151>`_) - Mention that ``app.make_handler`` is deprecated, recommend to use runners API instead. (`3157 <https://github.com/aio-libs/aiohttp/pull/3157>`_) Deprecations and Removals ------------------------- - Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 <https://github.com/aio-libs/aiohttp/pull/2826>`_) - Drop ``reader`` parameter from ``request.multipart()``. (`3090 <https://github.com/aio-libs/aiohttp/pull/3090>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/aiohttp - Changelog: https://pyup.io/changelogs/aiohttp/ - Repo: https://github.com/aio-libs/aiohttp </details>

socketpair · 2019-01-05T11:25:24Z

aiohttp/connector.py

@@ -895,6 +895,11 @@ def _get_fingerprint(self, req):
        transport, proto = await self._create_direct_connection(
            proxy_req, [], timeout, client_error=ClientProxyConnectionError)

+        # Many HTTP proxies has buggy keepalive support.  Let's not


@asvetlov do you have exact references which proxies do not support keepalive correctly ?

Rejecting keepalive will enlarge latencies, and make problems with frequent requests due to TIME_WAIT and inability to reuse TCP ports.

asvetlov · 2019-01-05T11:35:29Z

I had a bad experience in my previous job.
We used paid proxy fleets from different companies to mask our crawler agents.
Sorry, cannot provide more info.

webknjaz · 2019-01-05T11:38:55Z

Why not have a switch for it, for users who want to risk it?

socketpair · 2019-01-05T13:11:35Z

I vote for the switch, with default which ENABLES keep-alive.

asvetlov · 2019-01-05T13:15:45Z

PR is welcome!

lock · 2020-01-05T14:23:46Z

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.

asvetlov added 2 commits June 11, 2018 16:48

Disable keep-alive when working with proxy

b8830bc

Add changenote

3eba4b3

asvetlov merged commit 8b9924f into master Jun 12, 2018

asvetlov deleted the force-close-proxy branch June 12, 2018 08:39

asvetlov added a commit that referenced this pull request Jun 12, 2018

[3.3] Disable keep-alive when working with proxy (#3070)

81fd968

* Disable keep-alive when working with proxy * Add changenote (cherry picked from commit 8b9924f) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

dependencies bot mentioned this pull request Jun 13, 2018

aiohttp versions available: 3.3.2 Harmon758/Harmonbot#186

Closed

dependencies bot mentioned this pull request Aug 26, 2018

aiohttp versions available: 3.4.0 Harmon758/Harmonbot#301

Closed

socketpair reviewed Jan 5, 2019

View reviewed changes

lock bot added the outdated label Jan 5, 2020

lock bot locked as resolved and limited conversation to collaborators Jan 5, 2020

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Jan 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable keep-alive when working with proxy #3070

Disable keep-alive when working with proxy #3070

asvetlov commented Jun 11, 2018

socketpair commented Jun 11, 2018

asvetlov commented Jun 11, 2018

socketpair commented Jun 11, 2018

asvetlov commented Jun 11, 2018

codecov-io commented Jun 11, 2018

socketpair Jan 5, 2019

asvetlov commented Jan 5, 2019

webknjaz commented Jan 5, 2019

socketpair commented Jan 5, 2019

asvetlov commented Jan 5, 2019

lock bot commented Jan 5, 2020

Disable keep-alive when working with proxy #3070

Disable keep-alive when working with proxy #3070

Conversation

asvetlov commented Jun 11, 2018

socketpair commented Jun 11, 2018

asvetlov commented Jun 11, 2018

socketpair commented Jun 11, 2018

asvetlov commented Jun 11, 2018

codecov-io commented Jun 11, 2018

Codecov Report

socketpair Jan 5, 2019

Choose a reason for hiding this comment

asvetlov commented Jan 5, 2019

webknjaz commented Jan 5, 2019

socketpair commented Jan 5, 2019

asvetlov commented Jan 5, 2019

lock bot commented Jan 5, 2020