aiohttp request exceptions can't be caught sometimes when encountering bad http message

[voidfyoo]

Long story short

When encountering bad http message, aiohttp request exceptions can't be caught sometimes.

For example, I use code like below to detect if a proxy is working:

import sys
import asyncio
import logging

import aiohttp


DETECT_WEBSITE = 'httpbin.org'


async def proxy_req(proxy_url):
    if proxy_url.startswith('https://'):
        protocol = 'https'
    else:
        protocol = 'http'
    detect_url = f'{protocol}://{DETECT_WEBSITE}/get'
    proxy_url = proxy_url.replace('https://', 'http://')
    print('Detect url:', detect_url)
    print('Proxy url:', proxy_url)
    try:
        async with aiohttp.ClientSession() as session:
            async with session.get(detect_url,
                                   proxy=proxy_url,
                                   headers={'User-Agent': 'Mozilla/5.0'},
                                   timeout=10) as resp:
                text = await resp.text()
                print('Response text:')
                print(text)
    except Exception as exc:
        logging.error(exc)


if __name__ == '__main__':
    proxy_url = sys.argv[1]
    loop = asyncio.get_event_loop()
    loop.run_until_complete(proxy_req(proxy_url))

Expected behaviour

In the above code, I tried to catch all exceptions when doing request, so if a request exception happened, it should always be logged normally.

Actual behaviour

When I detect some broken proxies, for that proxy, sometimes the exception can be caught normally and logged, but sometimes the exception is not caught but are thrown directly.

For example, detect the broken proxy http://218.106.205.145:8080 ( When using this broken proxy to doing requests, it will return two different groups of reponse headers ), the output may look like below ( The first execution thrown exception, the second execution caught exception and logged ):

✗ python test.py http://218.106.205.145:8080
Detect url: http://httpbin.org/get
Proxy url: http://218.106.205.145:8080
Exception in callback None()
handle: <Handle cancelled>
Traceback (most recent call last):
  File "/Users/xxx/Coding/zzz/venv/lib/python3.6/site-packages/aiohttp/client_proto.py", line 161, in data_received
    messages, upgraded, tail = self._parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 297, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message='invalid constant string'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/selector_events.py", line 730, in _read_ready
    self._protocol.data_received(data)
  File "/Users/xxx/Coding/zzz/venv/lib/python3.6/site-packages/aiohttp/client_proto.py", line 177, in data_received
    self.transport.close()
AttributeError: 'NoneType' object has no attribute 'close'
Response text:
HTTP/1.1 500 OK
Date: Mon, 28 May 2018 09:43:07 GMT
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset="UTF-8"
Content-Length: 2173
Accep

✗ python test.py http://218.106.205.145:8080
Detect url: http://httpbin.org/get
Proxy url: http://218.106.205.145:8080
ERROR:root:400, message='invalid constant string'

Steps to reproduce

Run the code above to detect the broken proxy like http://218.106.205.145:8080

Your environment

aiohttp 3.2.1
Mac OS X
aiohttp client

[asvetlov]

Broken proxy can close connection at any place.
I can fix AttributeError but unfortunately you can get an empty response body sometimes.
The second situation cannot be detected by aiohttp itself.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new