Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bring in a string type for OS APIs #55

Open
alaviss opened this issue Oct 18, 2023 · 0 comments
Open

Bring in a string type for OS APIs #55

alaviss opened this issue Oct 18, 2023 · 0 comments
Labels
enhancement New feature or request windows Issues/PRs related to Windows support

Comments

@alaviss
Copy link
Owner

alaviss commented Oct 18, 2023

On Windows the native encoding is UTF-16. This poses a challenge where typical Nim strings has to converted to UTF-16 before they can be processed.

Right now, the conversion is done using WideCharToMultiByte and its counterpart, which silently drops invalid characters by normalizing them into � (U+FFFD -- Unicode Replacement Character). Furthermore, Windows itself doesn't deal in strict UTF-16, but rather WTF-16 (strings of arbitrary 16-bit codepoints) as a historical artifact.

All of this means that right now, sys operations on Windows have unexpected behaviors when the input string is not well-formed UTF-8 or when the output is not well-formed UTF-16.

To handle this, add a new string type called strings.Native (name subjected to change) that is in OS encoding and provide platform-specific conversion into their native code points.

An open question is to figure out what to do about strings.Without[T]: Whether to have it dealt in strings.Native or to create a new strings.WithoutNative[T].

@alaviss alaviss added enhancement New feature or request windows Issues/PRs related to Windows support labels Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request windows Issues/PRs related to Windows support
Projects
None yet
Development

No branches or pull requests

1 participant