UnicodeDecodeError when deserializing DynamoDB response

[adamrothman]

Problem

DynamoDB::Query operations fail due to a UnicodeDecodeError during response deserialization:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

An identical query succeeds using regular botocore.

Discussion

0x8b (139) is outside of the ASCII range and based on some cursory Googling, appears to indicate that the response is gzipped:

• https://stackoverflow.com/questions/11072705/twitter-trends-api-unicodedecodeerror-utf8-codec-cant-decode-byte-0x8b-in-po
• 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte ytdl-org/youtube-dl#366
• https://stackoverflow.com/questions/9445627/urllib2-opener-providing-wrong-charset

Related to #258?

Minimal reproducer

import asyncio
from pprint import pprint

import aiobotocore
import botocore.session


QUERY_KWARGS = {
    'ExpressionAttributeNames': {
        '#key': 'key',
        '#seq': 'seq',
        '#stream': 'stream',
        '#timestamp': 'timestamp',
        '#value': 'value',
    },
    'ExpressionAttributeValues': {
        ':stream': {'S': 'channel/1139320d29cb42e9813be24d363bc464/content'},
    },
    'IndexName': 'seq',
    'KeyConditionExpression': '#stream = :stream',
    'ProjectionExpression': '#seq, #key, #value, #timestamp',
    'ScanIndexForward': False,
    'TableName': 'flow-orbit-stage-events',
}


def query_botocore():
    session = botocore.session.get_session()

    ddb = session.create_client('dynamodb')
    paginator = ddb.get_paginator('query')
    page_iterator = paginator.paginate(**QUERY_KWARGS)

    for page in page_iterator:
        pprint(page)


async def query_aiobotocore():
    session = aiobotocore.get_session()

    async with session.create_client('dynamodb') as ddb:
        paginator = ddb.get_paginator('query')
        page_iterator = paginator.paginate(**QUERY_KWARGS)

        async for page in page_iterator:
            pprint(page)


if __name__ == '__main__':
    print('Running query with vanilla botocore')
    query_botocore()

    print()

    print('Running query with aiobotocore')
    asyncio.get_event_loop().run_until_complete(query_aiobotocore())

Result

$ python3 repro.py
Running query with vanilla botocore
{'Count': 6,
 'Items': [{'key': {'S': 'message/2a43dd8b854742bab50699ac6b61a341/reaction/947c284d507443b99802407a56b8e88b'},
            'seq': {'N': '7'},
            'timestamp': {'N': '1505153725.325'},
            'value': {'S': 'blah blah blah'}},

           ... omitted for brevity ...

 ],
 'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',
                                      'content-length': '4120',
                                      'content-type': 'application/x-amz-json-1.0',
                                      'date': 'Tue, 14 Nov 2017 20:43:06 GMT',
                                      'server': 'Server',
                                      'x-amz-crc32': '4255221333',
                                      'x-amzn-requestid': 'FCHVU5PNEKM7KL8BO1P444F8R7VV4KQNSO5AEMVJF66Q9ASUAAJG'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'FCHVU5PNEKM7KL8BO1P444F8R7VV4KQNSO5AEMVJF66Q9ASUAAJG',
                      'RetryAttempts': 0},
 'ScannedCount': 6}

Running query with aiobotocore
Traceback (most recent call last):
  File "repro.py", line 58, in <module>
    loop.run_until_complete(demo_aiobotocore())
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "repro.py", line 47, in demo_aiobotocore
    async for page in page_iterator:
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py", line 213, in coro
    res = yield from res
  File "/usr/local/lib/python3.6/site-packages/aiobotocore/paginate.py", line 41, in next_page
    response = yield from self._make_request(self._current_kwargs)
  File "/usr/local/lib/python3.6/site-packages/aiobotocore/client.py", line 80, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 249, in _send_request
    request, operation_model, attempts)
  File "/usr/local/lib/python3.6/site-packages/aiobotocore/endpoint.py", line 341, in _get_response
    response_dict, operation_model.output_shape)
  File "/usr/local/lib/python3.6/site-packages/botocore/parsers.py", line 212, in parse
    parsed = self._do_parse(response, shape)
  File "/usr/local/lib/python3.6/site-packages/botocore/parsers.py", line 597, in _do_parse
    original_parsed = self._parse_body_as_json(response['body'])
  File "/usr/local/lib/python3.6/site-packages/botocore/parsers.py", line 579, in _parse_body_as_json
    body = body_contents.decode(self.DEFAULT_ENCODING)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Environment

• Python 3.6
• aiobotocore 0.5.0
• aiohttp 2.3.2
• botocore 1.7.40

[adamrothman]

Both queries work as expected if I drop down to aiobotocore==0.4.5 and botocore==1.7.4.

[thehesiod]

see convo here: #430, try reverting the identity line and report back, thanks!

[jettify]

i have pr for that will merge soon and release new version

[jettify]

please try https://pypi.python.org/pypi/aiobotocore/0.5.1 feel free to open new ticket

[adamrothman]

Fixed, thank you @jettify!