Regression in parsing empty headers #3218

popravich · 2018-08-27T10:39:30Z

What do these changes do?

The v3.4 release bring regression in headers parsing:
empty header values are ignored and such empty headers get concatenated to first non-empty.

This PR just adds a test to recreate the issue #3219.

popravich · 2018-08-27T10:47:27Z

The issue is pretty severe:

#       assert raw_headers == ((b'X-Empty', b''),
#                              (b'Content-Length', b'0'),
#                              (b'Content-Type', b'application/octet-stream'),
#                              (b'Date', mock.ANY),
#                              (b'Server', mock.ANY))
+ ((b'X-Empty', b''),
- ((b'X-EmptyContent-Length', b'0'),
? ^   -------
+  (b'Content-Length', b'0'),
? ^
(b'Content-Type', b'application/octet-stream'),
-  (b'Date', b'Mon, 27 Aug 2018 10:42:00 GMT'),
-  (b'Server', b'Python/3.6 aiohttp/3.4.0'))
+  (b'Date', <ANY>),
+  (b'Server', <ANY>))

webknjaz · 2018-08-27T12:05:27Z

@popravich I assume it might be caused by #3209 if reproducible only with C-extensions enabled.

popravich · 2018-08-27T12:15:20Z

Nope, here the changeset that caused regression
https://github.com/aio-libs/aiohttp/pull/3095/files#diff-e1491b4a1efffe370338eece8f912848L144

webknjaz · 2018-08-27T12:56:05Z

btw why not fix master first?

popravich · 2018-08-27T13:01:58Z

I didn't want to play with all that cherry-pick-ing

webknjaz · 2018-08-27T14:32:12Z

CHANGES/3219.bugfix

@@ -0,0 +1 @@
+Fix empty header parsing regression.


You don't need this duplication.

webknjaz · 2018-08-27T14:32:52Z

But everything should go to master anyway. And better to do it first. It's a classical flow.

codecov-io · 2018-08-27T14:56:53Z

Codecov Report

Merging #3218 into 3.4 will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              3.4    #3218   +/-   ##
=======================================
  Coverage   98.07%   98.07%           
=======================================
  Files          43       43           
  Lines        7853     7853           
  Branches     1354     1354           
=======================================
  Hits         7702     7702           
  Misses         59       59           
  Partials       92       92

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f97bf0...3c98df8. Read the comment docs.

popravich · 2018-08-28T09:52:57Z

Hey, @asvetlov can you take a look at this?

asvetlov · 2018-08-28T14:17:02Z

@popravich thank you very much!
I'll forward it to master after making a bugfix release.

popravich · 2018-08-28T14:18:25Z

Thanks

This PR updates [aiohttp](https://pypi.org/project/aiohttp) from **3.3.2** to **3.4.4**. <details> <summary>Changelog</summary> ### 3.4.4 ``` ================== - Fix installation from sources when compiling toolkit is not available (`3241 <https://github.com/aio-libs/aiohttp/pull/3241>`_) ``` ### 3.4.3 ``` ================== - Add ``app.pre_frozen`` state to properly handle startup signals in sub-applications. (`3237 <https://github.com/aio-libs/aiohttp/pull/3237>`_) ``` ### 3.4.2 ``` ================== - Fix ``iter_chunks`` type annotation (`3230 <https://github.com/aio-libs/aiohttp/pull/3230>`_) ``` ### 3.4.1 ``` ================== - Fix empty header parsing regression. (`3218 <https://github.com/aio-libs/aiohttp/pull/3218>`_) - Fix BaseRequest.raw_headers doc. (`3215 <https://github.com/aio-libs/aiohttp/pull/3215>`_) - Fix documentation building on ReadTheDocs (`3221 <https://github.com/aio-libs/aiohttp/pull/3221>`_) ``` ### 3.4.0 ``` ================== Features -------- - Add type hints (`3049 <https://github.com/aio-libs/aiohttp/pull/3049>`_) - Add ``raise_for_status`` request parameter (`3073 <https://github.com/aio-libs/aiohttp/pull/3073>`_) - Add type hints to HTTP client (`3092 <https://github.com/aio-libs/aiohttp/pull/3092>`_) - Minor server optimizations (`3095 <https://github.com/aio-libs/aiohttp/pull/3095>`_) - Preserve the cause when `HTTPException` is raised from another exception. (`3096 <https://github.com/aio-libs/aiohttp/pull/3096>`_) - Add `close_boundary` option in `MultipartWriter.write` method. Support streaming (`3104 <https://github.com/aio-libs/aiohttp/pull/3104>`_) - Added a ``remove_slash`` option to the ``normalize_path_middleware`` factory. (`3173 <https://github.com/aio-libs/aiohttp/pull/3173>`_) - The class `AbstractRouteDef` is importable from `aiohttp.web`. (`3183 <https://github.com/aio-libs/aiohttp/pull/3183>`_) Bugfixes -------- - Prevent double closing when client connection is released before the last ``data_received()`` callback. (`3031 <https://github.com/aio-libs/aiohttp/pull/3031>`_) - Make redirect with `normalize_path_middleware` work when using url encoded paths. (`3051 <https://github.com/aio-libs/aiohttp/pull/3051>`_) - Postpone web task creation to connection establishment. (`3052 <https://github.com/aio-libs/aiohttp/pull/3052>`_) - Fix ``sock_read`` timeout. (`3053 <https://github.com/aio-libs/aiohttp/pull/3053>`_) - When using a server-request body as the `data=` argument of a client request, iterate over the content with `readany` instead of `readline` to avoid `Line too long` errors. (`3054 <https://github.com/aio-libs/aiohttp/pull/3054>`_) - fix `UrlDispatcher` has no attribute `add_options`, add `web.options` (`3062 <https://github.com/aio-libs/aiohttp/pull/3062>`_) - correct filename in content-disposition with multipart body (`3064 <https://github.com/aio-libs/aiohttp/pull/3064>`_) - Many HTTP proxies has buggy keepalive support. Let's not reuse connection but close it after processing every response. (`3070 <https://github.com/aio-libs/aiohttp/pull/3070>`_) - raise 413 "Payload Too Large" rather than raising ValueError in request.post() Add helpful debug message to 413 responses (`3087 <https://github.com/aio-libs/aiohttp/pull/3087>`_) - Fix `StreamResponse` equality, now that they are `MutableMapping` objects. (`3100 <https://github.com/aio-libs/aiohttp/pull/3100>`_) - Fix server request objects comparison (`3116 <https://github.com/aio-libs/aiohttp/pull/3116>`_) - Do not hang on `206 Partial Content` response with `Content-Encoding: gzip` (`3123 <https://github.com/aio-libs/aiohttp/pull/3123>`_) - Fix timeout precondition checkers (`3145 <https://github.com/aio-libs/aiohttp/pull/3145>`_) Improved Documentation ---------------------- - Add a new FAQ entry that clarifies that you should not reuse response objects in middleware functions. (`3020 <https://github.com/aio-libs/aiohttp/pull/3020>`_) - Add FAQ section "Why is creating a ClientSession outside of an event loop dangerous?" (`3072 <https://github.com/aio-libs/aiohttp/pull/3072>`_) - Fix link to Rambler (`3115 <https://github.com/aio-libs/aiohttp/pull/3115>`_) - Fix TCPSite documentation on the Server Reference page. (`3146 <https://github.com/aio-libs/aiohttp/pull/3146>`_) - Fix documentation build configuration file for Windows. (`3147 <https://github.com/aio-libs/aiohttp/pull/3147>`_) - Remove no longer existing lingering_timeout parameter of Application.make_handler from documentation. (`3151 <https://github.com/aio-libs/aiohttp/pull/3151>`_) - Mention that ``app.make_handler`` is deprecated, recommend to use runners API instead. (`3157 <https://github.com/aio-libs/aiohttp/pull/3157>`_) Deprecations and Removals ------------------------- - Drop ``loop.current_task()`` from ``helpers.current_task()`` (`2826 <https://github.com/aio-libs/aiohttp/pull/2826>`_) - Drop ``reader`` parameter from ``request.multipart()``. (`3090 <https://github.com/aio-libs/aiohttp/pull/3090>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/aiohttp - Changelog: https://pyup.io/changelogs/aiohttp/ - Repo: https://github.com/aio-libs/aiohttp </details>

lock · 2019-10-28T02:04:18Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new

Add a empty header value parsing test (failing test)

3e12a2c

popravich added the bug label Aug 27, 2018

popravich requested a review from asvetlov August 27, 2018 10:39

popravich added client server labels Aug 27, 2018

popravich changed the title ~~Add a empty header value parsing test (failing test)~~ Regression in parsing empty headers Aug 27, 2018

webknjaz mentioned this pull request Aug 27, 2018

Header parsing regression in 3.4 w/ empty headers #3219

Closed

fix empty header value parsing

3bd5206

popravich force-pushed the empty_header_parsing branch from 2104d7a to 3bd5206 Compare August 27, 2018 12:56

add one more related issue to CHANGES

0f1520b

webknjaz reviewed Aug 27, 2018

View reviewed changes

CHANGES/3219.bugfix Outdated

@@ -0,0 +1 @@

Fix empty header parsing regression.

Copy link

Member

webknjaz Aug 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need this duplication.

drop duplicate news fragment

3c98df8

asvetlov approved these changes Aug 28, 2018

View reviewed changes

asvetlov merged commit 470ccd0 into 3.4 Aug 28, 2018

asvetlov deleted the empty_header_parsing branch August 28, 2018 14:16

dependencies bot mentioned this pull request Aug 29, 2018

aiohttp versions available: 3.4.1 Harmon758/Harmonbot#305

Closed

lock bot added the outdated label Oct 28, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 28, 2019

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Oct 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression in parsing empty headers #3218

Regression in parsing empty headers #3218

popravich commented Aug 27, 2018 •

edited by webknjaz

Loading

popravich commented Aug 27, 2018

webknjaz commented Aug 27, 2018

popravich commented Aug 27, 2018

webknjaz commented Aug 27, 2018

popravich commented Aug 27, 2018

webknjaz Aug 27, 2018

webknjaz commented Aug 27, 2018

codecov-io commented Aug 27, 2018 •

edited

Loading

popravich commented Aug 28, 2018

asvetlov commented Aug 28, 2018

popravich commented Aug 28, 2018

lock bot commented Oct 28, 2019

Regression in parsing empty headers #3218

Regression in parsing empty headers #3218

Conversation

popravich commented Aug 27, 2018 • edited by webknjaz Loading

What do these changes do?

popravich commented Aug 27, 2018

webknjaz commented Aug 27, 2018

popravich commented Aug 27, 2018

webknjaz commented Aug 27, 2018

popravich commented Aug 27, 2018

webknjaz Aug 27, 2018

Choose a reason for hiding this comment

webknjaz commented Aug 27, 2018

codecov-io commented Aug 27, 2018 • edited Loading

Codecov Report

popravich commented Aug 28, 2018

asvetlov commented Aug 28, 2018

popravich commented Aug 28, 2018

lock bot commented Oct 28, 2019

popravich commented Aug 27, 2018 •

edited by webknjaz

Loading

codecov-io commented Aug 27, 2018 •

edited

Loading