You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On Windows the native encoding is UTF-16. This poses a challenge where typical Nim strings has to converted to UTF-16 before they can be processed.
Right now, the conversion is done using WideCharToMultiByte and its counterpart, which silently drops invalid characters by normalizing them into � (U+FFFD -- Unicode Replacement Character). Furthermore, Windows itself doesn't deal in strict UTF-16, but rather WTF-16 (strings of arbitrary 16-bit codepoints) as a historical artifact.
All of this means that right now, sys operations on Windows have unexpected behaviors when the input string is not well-formed UTF-8 or when the output is not well-formed UTF-16.
To handle this, add a new string type called strings.Native (name subjected to change) that is in OS encoding and provide platform-specific conversion into their native code points.
An open question is to figure out what to do about strings.Without[T]: Whether to have it dealt in strings.Native or to create a new strings.WithoutNative[T].
The text was updated successfully, but these errors were encountered:
On Windows the native encoding is UTF-16. This poses a challenge where typical Nim strings has to converted to UTF-16 before they can be processed.
Right now, the conversion is done using WideCharToMultiByte and its counterpart, which silently drops invalid characters by normalizing them into � (U+FFFD -- Unicode Replacement Character). Furthermore, Windows itself doesn't deal in strict UTF-16, but rather WTF-16 (strings of arbitrary 16-bit codepoints) as a historical artifact.
All of this means that right now,
sys
operations on Windows have unexpected behaviors when the input string is not well-formed UTF-8 or when the output is not well-formed UTF-16.To handle this, add a new string type called
strings.Native
(name subjected to change) that is in OS encoding and provide platform-specific conversion into their native code points.An open question is to figure out what to do about
strings.Without[T]
: Whether to have it dealt instrings.Native
or to create a newstrings.WithoutNative[T]
.The text was updated successfully, but these errors were encountered: