URL Params are decoded too often

[0xb35c]

Long story short

Passing a=%25s3 as a URL parameter causes an exception, saying "ValueError: Unallowed PCT %s3"
a=%25s3 is at some point decoded to a=%s3 and tried to parse as URL, which is of course invalid.

Expected behaviour

a=%25s3 should not be decoded to a=%s3

Actual behaviour

a=%25s3 is at some point decoded to a=%s3 and tried to parse as URL

Steps to reproduce

I debugged the whole issue. It seems the problem is in the file client_reqrep.py:77-81, where (if i understand correctly) the passed params (as dict) are merged with the params already in the URL.
Here is a code snippet showing the issue.

In [19]: URL('http://foo.bar').with_query(URL('http://bar.foo').with_query('a=%25').query)
Out[19]: URL('http://foo.bar/?a=')

In [20]: URL('http://bar.foo').with_query('a=%25')
Out[20]: URL('http://bar.foo/?a=%25')

In [21]: URL('http://bar.foo').with_query('a=%25').query
Out[21]: <MultiDictProxy('a': '%')>

Also here is some testcode:

import aiohttp
import asyncio

async def main(loop):
    params = {'key1': '%25s3', 'key2': 'value2'}
    with aiohttp.ClientSession() as session:
        async with session.get('http://127.0.0.1', params=params) as resp:
            print(resp.status)
            print(await resp.text())

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(loop))
    loop.close()

Your environment

• Linux-Arch 4.8.11-1-ARCH
• Python 3.5.2
• aiohttp 1.1.4

[asvetlov]

Thank you for report

[0xb35c]

No problem. Also in that context: I'd like to have a flag to mark URLs, data and params so that they won't be escaped or quoted. For testing web applications it might make sense to actually send a path or param like foo.bar/%s3?a=%s3

[wenxin-wang]

Confirmed with

• yarl==0.8.1
• aiohttp==1.2.0

For example:

from yarl import URL
str(URL("'"))
str(URL("%27"))
# Both evaluates to "'"

This is causing unexpected redirect loop in aiohttp.ClientSession.get when the server return redirection to the quoted URLs on seeing unquoted ones, in my case for example, http://starbounder.org/Bolt_O's

Right now URL's quoting could be bypassed by using urllib.parse.quote and the encoded keyword parameter in URL.__init__:

from yarl import URL
from urllib.parse import quote
str(URL(quote("'"), encoded=True))
# Evaluates to "%27"

[fafhrd91]

i think this is fixed with yarl 0.9.1

[fafhrd91]

@0xb35c @wenxin-wang could you check aiohttp master and yarl 0.9.1?

[wenxin-wang]

@fafhrd91 I removed the original venv and used

pip install git+https://github.com/KeepSafe/aiohttp.git
# aiohttp==1.3.0a0
# yarl==0.9.1

I still got the problem above.

from yarl import URL
str(URL("'"))
str(URL("%27"))
# Both evaluates to "'"

To be more specific, try

import asyncio as aio
import aiohttp
async def get_content(session, url):
    async with session.get(url) as res:
        await res.release()
        return res.history

async def main(loop):
    async with aiohttp.ClientSession(loop=loop) as session:
        return await get_content(session, "http://starbounder.org/Bolt_O's")

loop = aio.get_event_loop()
print(loop.run_until_complete(main(loop)))

You will get 10 redirections where the server sent redirect to http://starbounder.org/Bolt_O%27s but the client still visit the unquoted URL.

[lukkm]

Are there any updates on this thread? Also, is there any way to work around this for now? Like, send a request to http://bar.foo/?a=%25 without the client crashing.

[fafhrd91]

@lukkm you can use yarl.URL('http://bar.foo/?a=%25', encoded=True) as url

[vladarts]

Hello! I have same problem.

During redirect, Location header is
https://yandex.ru/showcaptcha?cc=1&retpath=https%3A//yandex.ru/search%3Ftext%3D%25D1%258F%2B%25D0%25B2%2B%25D1%2588%25D0%25BE%25D0%25BA%25D0%25B5_ae7b4fe85db2da139661590c878cf956&t=0/1490699030/593528fc0cbb55f78c768d1184d492c8&s=fbbbb021daa71c81162711feefe39e22

But really request goes to url
https://yandex.ru/showcaptcha?cc=1&retpath=https://yandex.ru/search?text%3D%25D1%258F%2B%25D0%25B2%2B%25D1%2588%25D0%25BE%25D0%25BA%25D0%25B5_ae7b4fe85db2da139661590c878cf956&t=0/1490699030/593528fc0cbb55f78c768d1184d492c8&s=fbbbb021daa71c81162711feefe39e22

on yandex.ru it is a problem

if patch aiohttp.client.ClientSession#_request method (use url = URL(url, encoded=True) instead of url = URL(url) ) - everything works OK

UPDATE

This code is working

async with self.session.get(url, allow_redirects=False) as _response:
    response = Selector(await _response.text())

if _response.status == 302:
    redirect_url = URL(_response.headers['Location'], encoded=True)
    async with self.session.get(redirect_url) as _response:
        response = Selector(await _response.text())

[fafhrd91]

added requote_redirect_url session attribute.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.