Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update aiohttp #425

Merged
merged 10 commits into from
Nov 9, 2017
Merged

Update aiohttp #425

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a19f3e5
update changes
jettify Nov 7, 2017
9a6ed0b
work
thehesiod Jul 18, 2017
839b737
add placeholders
thehesiod Jul 18, 2017
5829e6f
add changes text
thehesiod Jul 18, 2017
2a27b96
put this section back with updated comment
thehesiod Aug 2, 2017
7298c12
put back
thehesiod Aug 2, 2017
29ee0d4
update aiohttp
jettify Nov 7, 2017
05f2db7
flake8
jettify Nov 7, 2017
786a9a0
update setup.py
jettify Nov 9, 2017
579ba50
Merge branch 'master' into update-aiohttp
jettify Nov 9, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CHANGES.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
Changes
-------
0.4.6a0 (XXXX-XX-XX)
0.5.0 (XXXX-XX-XX)
^^^^^^^^^^^^^^^^^^
* fix #190
* Fix error botocore error checking #190
* Update supported botocore requirement to: >=1.7.28, <=1.7.40
* Bump aiohttp requirement to support compressed responses correctly #298

0.4.5 (2017-09-05)
^^^^^^^^^^^^^^^^^^
Expand Down
82 changes: 7 additions & 75 deletions aiobotocore/endpoint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
import aiohttp.http_exceptions
from aiohttp.client_proto import ResponseHandler
from aiohttp.helpers import CeilTimeout
from aiohttp import hdrs
from aiohttp.client import URL
from aiohttp.client_reqrep import ClientResponse
from botocore.endpoint import EndpointCreator, Endpoint, DEFAULT_TIMEOUT, \
Expand Down Expand Up @@ -170,45 +169,6 @@ def read(self):
return resp_msg, stream_reader


def _aiohttp_do_redirect(session, method, url, headers, data, resp):
# This is the redirect code from aiohttp, remove once
# https://github.com/aio-libs/aiobotocore/issues/267 is supported

# For 301 and 302, mimic IE, now changed in RFC
# https://github.com/kennethreitz/requests/pull/269

if (resp.status == 303 and
resp.method != hdrs.METH_HEAD) \
or (resp.status in (301, 302) and
resp.method == hdrs.METH_POST):
method = hdrs.METH_GET
data = None
if headers.get(hdrs.CONTENT_LENGTH):
headers.pop(hdrs.CONTENT_LENGTH)

r_url = (resp.headers.get(hdrs.LOCATION) or
resp.headers.get(hdrs.URI))
if r_url is None:
return None

r_url = URL(
r_url, encoded=not session.requote_redirect_url)

scheme = r_url.scheme
if scheme not in ('http', 'https', ''):
resp.close()
raise ValueError(
'Can redirect only to http or https')
elif not scheme:
r_url = url.join(r_url)

url = r_url
params = None
resp.release()

return method, url, headers, params, data


class AioEndpoint(Endpoint):
def __init__(self, host,
endpoint_prefix, event_emitter, proxies=None, verify=True,
Expand Down Expand Up @@ -259,23 +219,11 @@ def __init__(self, host,
conn_timeout=self._conn_timeout,
skip_auto_headers={'CONTENT-TYPE'},
response_class=ClientResponseProxy,
loop=self._loop)
loop=self._loop,
auto_decompress=False)

@asyncio.coroutine
def _request(self, method, url, headers, data):
# Note: When using aiobotocore with dynamodb, requests fail on crc32
# checksum computation as soon as the response data reaches ~5KB.
# When AWS response is gzip compressed:
# 1. aiohttp is automatically decompressing the data
# (http://aiohttp.readthedocs.io/en/stable/client.html#binary-response-content)
# 2. botocore computes crc32 on the uncompressed data bytes and fails
# cause crc32 has been computed on the compressed data
# The following line forces aws not to use gzip compression,
# if there is a way to configure aiohttp not to perform decompression,
# we can remove the following line and take advantage of
# aws gzip compression.
# See: https://github.com/aio-libs/aiohttp/issues/1992
headers['Accept-Encoding'] = 'identity'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jettify bug: we need to keep this identity until boto/botocore#1255 is fixed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, will bring back this lines.

headers_ = MultiDict(
(z[0], text_(z[1], encoding='utf-8')) for z in headers.items())

Expand All @@ -286,27 +234,11 @@ def _request(self, method, url, headers, data):
data = _IOBaseWrapper(data)

url = URL(url, encoded=True)

# See https://github.com/aio-libs/aiobotocore/issues/267 for details
for i in range(MAX_REDIRECTS):
resp = yield from self._aio_session.request(method, url=url,
headers=headers_,
data=data,
proxy=proxy,
timeout=None,
allow_redirects=False)

if resp.status in {301, 302, 303, 307}:
redir_arr = _aiohttp_do_redirect(self._aio_session, method,
url, headers, data, resp)

if redir_arr is None:
break

method, url, headers, params, data = redir_arr
else:
break

resp = yield from self._aio_session.request(method, url=url,
headers=headers_,
data=data,
proxy=proxy,
timeout=None)
return resp

@asyncio.coroutine
Expand Down
2 changes: 1 addition & 1 deletion requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ moto==0.4.31
pytest-cov==2.5.1
pytest==3.2.3
sphinx==1.6.5
aiohttp==2.2.5
aiohttp==2.3.2
botocore==1.7.40
multidict==3.3.2
wrapt==1.10.11
Expand Down
2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
# If adding requirements make sure to also add to requirements-dev.txt
install_requires = [
'botocore>=1.7.28, <=1.7.40',
'aiohttp>=2.0.4, <=2.3.0',
'aiohttp>=2.3.0, <=2.3.2',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Push <3.0.0 here.
All aiohttp 2.3 bugfixes should be backward compatible.
Honestly I don't expect many of them, maybe one or two with very minor changes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

'multidict>=2.1.4',
'wrapt>=1.10.10',
'packaging>=16.8',
Expand Down
9 changes: 4 additions & 5 deletions tests/test_patches.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,19 @@
# github.com/aio-libs/aiobotocore/pull/248
_READ_TIMEOUT_DIGESTS = {
# for our replacement of _factory
TCPConnector: {'a153acda1bfc891d01a56597f77b33afbd786d3d'},
TCPConnector: {'1e6a3c14ce703781253da5cfdc7847b4ae898234'},

# for its inheritance to DataQueue
ResponseHandler: {'1cfdb033cb4b4d586bbebf49bed3c2930c026809'},
ResponseHandler: {'5f11c28d0075e36dfec4705891f1c90248202ba4'},

# for our replacement of read()
DataQueue: {'8ad4d5df1d016547daea6389707bc656630582e5'},

# for our patch of _wait
StreamReader: {'dc5a5a33e86cedf2d2c8413d951d2274a79303f4'},
StreamReader: {'c0a9a31a8c3e550de5985ab642028983f709b37b'},

# for digging into _protocol ( 2.1.x, 2.2.x )
ClientResponse: {'d1e0c16dea4fe3426caa1e9b0dc9f5f1992d838e',
'bc374038ac3bfd7cc13dadb6aebbf0f67ebb7620'},
ClientResponse: {'1dc0008e88b3f5ec2f59f6f5f03fae601f4a011d'},
}

# These are guards to our main patches
Expand Down