percent-encode ' in queries of URLs with special schemes

[achristensen07]

This is needed for compatibility, and it makes some injection attacks harder by preventing ' from being sent to the server.

Chrome, Firefox, and Safari all have this behavior. I haven't tested Edge or IE. This can be verified as simply as this:
alert(new URL("http://host/pa'th?qu'ery#fra'gment"));
alert(new URL("asdf://host/pa'th?qu'ery#fra'gment"));

[TimothyGu]

In both Chrome and Firefox here, the ' in the second URL doesn't seem to be escaped...

[achristensen07]

Exactly. We need to add it to the list of characters to be escaped if the URL has a special scheme like http.

[TimothyGu]

Ah, sorry I misunderstood your original issue.

[annevk]

cc @valenting

[valenting]

I think this is a good move. Since it's the behaviour already present in Chrome and Firefox, the chance of breaking sites is very low. Let's do it.

[mgiuca]

I'm wondering why ' is singled out here for "is special".

Referring to the new language that was added:

byte is 0x27 (') and url is special

I tested query parsing for all ASCII characters in Chrome and Firefox:

Special URL:

new URL("http://example.com/? !\"$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~").search

Chrome:

"?%20!%22$%&%27()*+,-./0123456789:;%3C=%3E?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~"

Firefox:

"?%20!%22$%&%27()*+,-./0123456789:;%3C=%3E?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~"

(Both encode <0x21, 0x22, 0x27, 0x3C, 0x3E, >0x7E — agreement with each other and spec.)

Non-special URL:

new URL("ssh://example.com/? !\"$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~").search

Chrome:

"? !"$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~"

Firefox:

"?%20!"$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~"

Further experimentation shows that Chrome encodes <0x20 while Firefox encodes <0x21. No other ASCII characters are encoded.

So it seems that there's nothing special about ' here; none of these characters are encoded for special URLs. So this could just read: "byte is 0x22 ("), 0x23 (#), 0x27 ('), 0x3C (<), or 0x3E (>) and url is special" and would be "more accurate".

(Having said all this, it looks like the path and other components are similarly not very well encoded for non-special URLs so there isn't much point fixing this for query unless it's fixed everywhere.)