Handle urllib3 timeout errors

[danielgtaylor]

This is an attempt to fix an issue where a successful HTTP request is made
but while reading the raw low-level response we may get a urllib3 error
raised instead of the corresponding requests error.

The error presents itself when sending/receiving large files to/from S3
with the AWS CLI when on a bad network connection.

This makes sure that our retry handler is capable of handling this case
and adds a test for it.

Traceback sample:

Traceback (most recent call last):
  File ".../site-packages/awscli/customizations/s3/tasks.py", line 347, in __call__
    self._download_part()
  File ".../site-packages/awscli/customizations/s3/tasks.py", line 376, in _download_part
    self._queue_writes(body)
  File ".../site-packages/awscli/customizations/s3/tasks.py", line 407, in _queue_writes
    self._queue_writes_in_chunks(body, iterate_chunk_size)
  File ".../site-packages/awscli/customizations/s3/tasks.py", line 438, in _queue_writes_in_chunks
    current = body.read(iterate_chunk_size)
  File ".../site-packages/botocore/response.py", line 70, in read
    chunk = self._raw_stream.read(amt)
  File ".../site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 210, in read
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
ReadTimeoutError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Read timed out.

It's difficult to reproduce the issue and test the fix. Any ideas you have would be appreciated.

cc @kyleknap @jamesls

[coveralls]

Coverage remained the same at 94.5% when pulling 2e2fa64 on danielgtaylor:urllib3-timeout into 6a63e18 on boto:develop.

[jamesls]

Have you tested this against the real service? i.e a read timeout when reading from an s3 body during a GetObject (or even an aws s3 cp based on the traceback). I'm 99% sure this PR won't do anything, as the retry logic happens within the scope of make_request and we're way past that by the time we're queueing writes to the IO queue in the CLI.

[jamesls]

Closing, superseded by aws/aws-cli#1191. In general, the thing that makes the initial request and pulls from the stream is the thing that needs to own retrying the request on read timeouts. In this specific scenario, the AWS CLI is where the fix needed to be made.

[kushalmangtani]

I am using the boto3 python sdk for executing the GET operation. As I learned, you have to explicitly call read to get the entire object.
I realized that read() does not throws ClientError as mentioned in botocore exception handling. So, I have to catch all exception since read() method throws urllib3 exceptions (specifically ReadTimeoutError as in the above traceback)
Is there a better way to handle such botocore exceptions ?

Snippet of code and traceback:

  response = client.get_object(
                  Bucket=bucket_name, Key=object_name)
  response['Body'].read()
except ClientError as c:
  logger.error("Encoutered a client error exception")
# For catching response['Body'].read() exceptions.
except Exception as e:
  logger.error("Encountered a general exception")```

# If I remove Base Exception block, I see ReadTimeoutError. 

# Traceback
```File "/home/ubuntu/ObjectStorage/client-python/ost/client.py", line 124, in <module>
    main()
  File "/home/ubuntu/ObjectStorage/client-python/ost/client.py", line 99, in main
    results = client.get(args.bucket, args.key, args.size)
  File "/home/ubuntu/ObjectStorage/client-python/ost/provider/s3.py", line 176, in get
    response['Body'].read()
  File "/usr/local/lib/python3.5/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/local/lib/python3.5/dist-packages/botocore/vendored/requests/packages/urllib3/response.py", line 258, in read
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
botocore.vendored.requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='sthree-monitoring.s3.amazonaws.com', port=443): Read timed out```