Skip to content
This repository has been archived by the owner on Feb 15, 2024. It is now read-only.

UTF-8 not handled while writing #16

Closed
laanwj opened this issue Nov 5, 2015 · 1 comment · Fixed by #22
Closed

UTF-8 not handled while writing #16

laanwj opened this issue Nov 5, 2015 · 1 comment · Fixed by #22

Comments

@laanwj
Copy link
Contributor

laanwj commented Nov 5, 2015

UTF-8 is not handled while writing strings in json_escape. The result of this is that unicode sequences like '\u1234' will be expanded to their UTF-8 equivalent on a roundtrip, and re-encoded as multiple unicode characters \u00XX\u00XX... which is not correct.

This could be handled:

If the input/output is UTF-8 encoded, UTF-8 sequences in strings can be passed through on output without any processing (according to RFC 4627 "All Unicode characters may be placed within the quotation marks except for the characters that must be escaped: quotation mark, reverse solidus, and the control characters (U+0000 through U+001F).")

From RFC 4627:

JSON text SHALL be encoded in Unicode.  The default encoding is
UTF-8.
@apoelstra
Copy link

To be consistent with https://github.com/apoelstra/strason the behaviour should be:

  • For the appropriate escape characters (\n, \r, \b, \f, ", ) use those. (It's allowable to escape / as well but I don't.)
  • For isprint ASCII characters, pass through.
  • For everything else, encode as UTF-16BE and use \uXXXX.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants