Failure parsing querystring from windows client

[dspaeth-faber]

Hi,

I have a windows client which sends a query string url encoded but the character encoding is
not utf-8. I did open an issue within Qs. Unfortunately they say it's none of their business, only utf-8 is supported. querystring does also not work.

Is this to be considered a bug or bad luck with my client?

regards
dieter

The example:

qs = require('qs')

describe 'Qs', ->

  describe '#parse', ->
    it 'parses ucs2 from windows', (done) ->
      expect(unescape('%f6%e4%fc%e1%20')).to.be.like('öäüá ')
      expect(qs.parse('a=%f6%e4%fc%e1%20')).to.be.like( {'a': 'öäüá '})
      done()

[dougwilson]

Yes, absolutely only UTF-8 is accepted in the URL. The reason is two-fold: 1) only UTF-8 is allowed in urlencoded ocets in URLs because it's the URL specification and 2) there is no data contained in the URL to signify what encoding the data is in (and it's not necessarily, as it must be UTF-8).

It sounds like the client you are using is simply buggy or you may not be using it right.

[dougwilson]

In case it helps, this is the correct C# method to encode URL components with: http://msdn.microsoft.com/en-us/library/system.net.webutility.urlencode(v=vs.110).aspx and it uses UTF-8, of course.

[dspaeth-faber]

@dougwilson thanks for your help, but It doesn't help. The call comes from within Transact SQL a stored proc.

at 1) Querystrings don't need to be UTF-8, RFC's say nothing about encoding like RFC 3629, RFC 3986

Take a look here: http://www.motobit.com/util/url-decoder.asp
and try http://www.example.com?a=%f6%e4%fc%e1%20 it is decoded right.

[dougwilson]

Please read RFC 3986:

When a new URI scheme defines a component that represents textual
data consisting of characters from the Universal Character Set [UCS],
the data should first be encoded as octets according to the UTF-8
character encoding [STD63]; then only those octets that do not
correspond to characters in the unreserved set should be percent-
encoded. For example, the character A would be represented as "A",
the character LATIN CAPITAL LETTER A WITH GRAVE would be represented
as "%C3%80", and the character KATAKANA LETTER A would be represented
as "%E3%82%A2".

The http: URI scheme defines the path-abempty, query, and fragment components to consist of charasters from UCS and therefore they should be UTF-8 octets. Back a long time ago browsers and people were doing different things. These days are long gone and many (like the Node.js community) only support UTF-8 octets.

[dougwilson]

The end answer is, if you want it to work with express out-of-the-box, you need to fix your TSQL to send UTF-8. Otherwise, you can use your own query string parser.

To use your own custom query string parser, here is an example (requires express >= 4.7.0):

app.set('query parser', function (string) {
  return parseMyQueryString(string);
});

[dspaeth-faber]

Ok.