ISO-8859-1 encoded http headers

[ephes]

Hi,

gunicorn uses utf8 encoding for http response headers. I don't know
much about http standards, but this is probably not correct:

http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding

best regards,
Jochen

[benoitc]

The tests don't pass. unitests.mock don't pass. Can you fix it?

Also reading the new RFC 7230:

Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.

Not sure how to handle opaque data there. Also including such change we should also test if it works with some value given as unicode (which happen sometime in some countries...). Maybe we should have more tests there. Ideally we shouldn't transform anything there.

Note: we are converting here due to the silly way bytes, native strings have been managed between py2 and py3.

[jamadden]

Also including such change we should also test if it works with some value given as unicode (which happen sometime in some countries...).

Of course, according to the WSGI spec, that's not supposed to happen in Python 2. Headers are specified to be given as the "native string type", so they should already be bytes and applications that send unicode values are in non-compliance with the spec (I've seen middleware break due to a buggy application that had a unicode header value). Likewise under Python 3 (where the native string type is unicode) including non-latin-1-encodable data is also out of compliance with the spec, the HTTP spec this time, as well as the WSGI spec:

Do not be confused however: even if Python's str type is actually Unicode "under the hood", the content of native strings must still be translatable to bytes via the Latin-1 encoding!

So either case will enter implementation-defined behaviour and not be interoperable.

[tilgovi]

👍 to this change

[berkerpeksag]

latin1 -> latin-1. latin1 is an alias of latin-1.

[berkerpeksag]

Good catch, thanks! Could you please squash the commits?

[ephes]

Ok, squashed the commits :).

[berkerpeksag]

This change shouldn't be here :) See the original commit: 8de5eb9

In general, the patch LGTM except this, but I can take care of it if you don't have time.

Thanks!

[ephes]

Yup, this line was a leftover from an unintentional merge :/. Thanks for pointing it out - it's now removed.

[berkerpeksag]

Thanks!