Request via aiohttp proxy does not work (url is None) #1413

nodermann2 · 2016-11-20T14:07:52Z

Long story short

I tried to perform a request via http proxy, but the request via aiohtpp doesn't work, example is taken from official documentation, put a random proxy, and you get the same result
random proxy list http://hideme.ru/proxy-list/?ports=80#list
or
python proxy for test https://github.com/abhinavsingh/proxy.py

Expected behaviour

curl --proxy 163.172.177.100:80 -I http://httpbin.org/
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 20 Nov 2016 14:02:24 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12150
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Cache: MISS from apocas
X-Cache-Lookup: MISS from apocas:80
Via: 1.1 apocas (squid/3.3.8)
Connection: keep-alive

status 200

Actual behaviour

proxy response: status 400 - Invalid URL / Some aspect of the requested URL is incorrect.

Steps to reproduce

import asyncio
import aiohttp

async def go():
    async with aiohttp.ClientSession() as session:
        async with session.get("http://httpbin.org/",
                               proxy="http://163.172.177.100:80") as resp:
            print(resp.status)


loop = asyncio.get_event_loop()
loop.run_until_complete(go())
loop.close()

Your environment

tried on both python 3.5.2 and python 3.6 (latest)
aiohttp (1.1.5)

Debug

I turned on my proxy, look at the results
curl
2016-11-20 18:30:54,049 - INFO - pid:24748 - 127.0.0.1:33562 - b'HEAD' b'httpbin.org':80b'/' - b'200' b'OK' - 225 bytes
aiohttp:
2016-11-20 18:30:07,162 - INFO - pid:20625 - 127.0.0.1:33548 - b'GET' None:80b'/' - None None - 0 bytes

The text was updated successfully, but these errors were encountered:

pawelmhm · 2016-11-20T18:26:48Z

Yeah I can reproduce. Seems like MITM-proxy responds with following message:


            <html>
                <head>
                    <title>400 Bad Request</title>
                </head>
                <body>HttpError('Invalid HTTP request form (expected: absolute, got: relative)',)</body>
            </html>

and request does not go through. Bisecting suggests that first wrong commit is e81a478

git bisect start
# bad: [b65ecf48278c75f255871bae530904bc34b6f731] Add a FAQ chapter for deprecation policy
git bisect bad b65ecf48278c75f255871bae530904bc34b6f731
# good: [728bc35211b6ce3680643f182ba9367c31bbee0f] Update CHANGES
git bisect good 728bc35211b6ce3680643f182ba9367c31bbee0f
# bad: [380f2c0beb713f4a4086dbef8e2042b9a980d1f9] Merge branch 'master' of github.com:KeepSafe/aiohttp
git bisect bad 380f2c0beb713f4a4086dbef8e2042b9a980d1f9
# bad: [e94b65e4c6d2d381efd4fafee2d51a6534915315] Pin flake8 to latest version 3.0.4 (#1269)
git bisect bad e94b65e4c6d2d381efd4fafee2d51a6534915315
# good: [20ffd6b36977cb344a258bc6ac14f88955dc669b] Merge branch '1.0'
git bisect good 20ffd6b36977cb344a258bc6ac14f88955dc669b
# bad: [8b849540810538263170cee1c484b9e37717f4d3] Pin async-timeout to latest version 1.0.0 (#1235)
git bisect bad 8b849540810538263170cee1c484b9e37717f4d3
# good: [f3d066eb050a64c428fd3bef5301d8ae4f9d35ad] Merge branch 'master' of github.com:KeepSafe/aiohttp
git bisect good f3d066eb050a64c428fd3bef5301d8ae4f9d35ad
# bad: [a93d9c2e564fbe7f0461f2e9063ccab76c723a39] Close idle keep-alive connections on shutdown (#1222)
git bisect bad a93d9c2e564fbe7f0461f2e9063ccab76c723a39
# bad: [e81a4781d80b614da572c2c18635831ad024126b] Use yarl in client API (#1217)
git bisect bad e81a4781d80b614da572c2c18635831ad024126b
# good: [0d41462675c9781a89e1bd3b41ee16be0c183ea5] Use 1001 Going-Away code in websocket shutdown example (#1215) (#1216)
git bisect good 0d41462675c9781a89e1bd3b41ee16be0c183ea5
# first bad commit: [e81a4781d80b614da572c2c18635831ad024126b] Use yarl in client API (#1217)

* fix bug that broke HTTP proxy support * add functional tests via mitmdump fixes aio-libs#1413

nodermann2 · 2016-11-21T21:49:59Z

There are also suspicions that the squid proxy does not properly handle redirect requests or aiohttp does not know how to read some of the headers

code

import asyncio
import aiohttp


async def go():
    async with aiohttp.ClientSession() as session:
        async with session.get("http://httpbin.org/relative-redirect/2",
                               proxy="http://ip:port") as resp:
            print(resp.status)


loop = asyncio.get_event_loop()
loop.run_until_complete(go())
loop.close()

unknown proxy software - status 200

headers

<CIMultiDictProxy('Server': 'nginx', 'Date': 'Mon, 21 Nov 2016 22:08:29 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '0', 'Location': '/relative-redirect/1', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Proxy-Connection': 'Keep-Alive')>

squid proxy - status 400

headers

<CIMultiDictProxy('Server': 'nginx', 'Date': 'Mon, 21 Nov 2016 22:07:42 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '0', 'Location': '/relative-redirect/1', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'X-Cache': 'MISS from adsl', 'X-Cache-Lookup': 'MISS from adsl:29842', 'Connection': 'keep-alive')>

asvetlov · 2016-11-24T20:31:31Z

I believe it's solved by @1421

nodermann2 · 2016-11-25T00:16:36Z

Yes it works correctly, thank you very much.

lock · 2019-10-29T04:02:47Z

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.

nodermann2 changed the title ~~Request via aiohttp proxy does not work correctly~~ Request via aiohttp proxy does not work Nov 20, 2016

nodermann2 changed the title ~~Request via aiohttp proxy does not work~~ Request via aiohttp proxy does not work (url is None) Nov 20, 2016

pawelmhm added a commit to pawelmhm/aiohttp that referenced this issue Nov 20, 2016

fix HTTP proxying

078d852

* fix bug that broke HTTP proxy support * add functional tests via mitmdump fixes aio-libs#1413

pawelmhm mentioned this issue Nov 20, 2016

fix HTTP proxying #1415

Closed

5 tasks

pawelmhm added a commit to pawelmhm/aiohttp that referenced this issue Nov 20, 2016

fix HTTP proxying

ac8b43f

* fix bug that broke HTTP proxy support * add functional tests via mitmdump fixes aio-libs#1413

kserhii added a commit to kserhii/aiohttp that referenced this issue Nov 22, 2016

Fix bugs with client proxy: target path, host with port (aio-libs#1413)

3f283c6

kserhii mentioned this issue Nov 23, 2016

Proxy fix #1421

Merged

5 tasks

asvetlov closed this as completed Nov 24, 2016

lock bot added the outdated label Oct 29, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Request via aiohttp proxy does not work (url is None) #1413

Request via aiohttp proxy does not work (url is None) #1413

nodermann2 commented Nov 20, 2016 •

edited

Loading

pawelmhm commented Nov 20, 2016 •

edited

Loading

nodermann2 commented Nov 21, 2016 •

edited

Loading

asvetlov commented Nov 24, 2016

nodermann2 commented Nov 25, 2016

lock bot commented Oct 29, 2019

Request via aiohttp proxy does not work (url is None) #1413

Request via aiohttp proxy does not work (url is None) #1413

Comments

nodermann2 commented Nov 20, 2016 • edited Loading

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

Debug

pawelmhm commented Nov 20, 2016 • edited Loading

nodermann2 commented Nov 21, 2016 • edited Loading

code

unknown proxy software - status 200

headers

squid proxy - status 400

headers

asvetlov commented Nov 24, 2016

nodermann2 commented Nov 25, 2016

lock bot commented Oct 29, 2019

nodermann2 commented Nov 20, 2016 •

edited

Loading

pawelmhm commented Nov 20, 2016 •

edited

Loading

nodermann2 commented Nov 21, 2016 •

edited

Loading