Conform to RFC3986 - do not include url fragments in requests

[bmuller]

What do these changes do?

This strips fragments from URLs before making requests. This is the behavior of all major browsers and is described in RFC3986 Section 3.5.

Are there changes in behavior for the user?

Nope.

Related issue number

None.

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes
• Add yourself to CONTRIBUTORS.txt
• Add a new entry to CHANGES.rst
  • Choose any open position to avoid merge conflicts with other PRs.
  • Add a link to the issue you are fixing (if any) using #isuue_number format at the end of changelog message. Use Pull Request number if there are no issues for PR or PR covers the issue only partially.

[codecov-io]

Current coverage is 98.40% (diff: 100%)

Merging #1174 into master will increase coverage by <.01%

@@             master      #1174   diff @@
==========================================
  Files            29         29          
  Lines          6514       6515     +1   
  Methods           0          0          
  Messages          0          0          
  Branches       1092       1092          
==========================================
+ Hits           6410       6411     +1   
  Misses           54         54          
  Partials         50         50          

Powered by Codecov. Last update da03eef...9d08b9b

[bmuller]

As an example of why this is needed, right now aiohttp can't be used to download any post pages from tumblr.com because their server goes into an infinite redirect loop when a fragment is included. Here's an example script:

import aiohttp
import asyncio

async def fetch(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            print(resp.history)

url = "http://best-of-tumblr.tumblr.com/post/150420362886"
asyncio.get_event_loop().run_until_complete(fetch(url))

The history that's output is just a list max_redirects long showing an infinite redirect loop. If you use my fork, then there's only one redirect and everything works fine.

[asvetlov]

Cool!
We already drop fragments on server side redirection, supporting the feature on client is very important addition.
Please add yourself to CONTRIBUTORS.txt and file a new entry to CHANGES.rst

[bmuller]

Done! Let me know if you need anything else.

[asvetlov]

Nothing more -- your PR is very clean and obvious.
Thanks!

[bmuller]

Thanks so much for the quick turnaround!

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.