Emoji support

[arpit]

Current parser crashes when trying to load a yaml document with emojis in it.

    var emoji = "😃";
    String s = "YAML:  ${emoji} YAML Ain't Markup Language";
    var doc = loadYaml(s);
    print(doc['YAML']);

Error:

Error on line 1, column 8: Unexpected character.
  ╷
1 │ YAML:  😃 YAML Ain't Markup Language
  │        ^
  ╵

package:yaml/src/parser.dart 50:7    Parser.parse
package:yaml/src/loader.dart 165:37  Loader._loadMapping
package:yaml/src/loader.dart 86:16   Loader._loadNode
package:yaml/src/loader.dart 62:20   Loader._loadDocument

[fawazhussain]

flutter_emoji is a package which can parse emojis and can be stored as strings for later reference.
https://pub.dev/packages/flutter_emoji

[kevmoo]

I just hit this, too!

[kevmoo]

It seems that the yaml package restricts the INPUT parsing to the same restrictions as the YAML spec defines for OUTPUT, which may be the problem.

See https://yaml.org/spec/1.2/spec.html#id2770814

[littoy]

Any plan to fix this bug?

[kevmoo]

PR welcome!

[AdrKacz]

Hello 😊 Thank you for this package!

Any updates on that? It would great to load emojis!

**Edit: ** I just figured out that everything works fine if I double quote my emojis (YAML: Hello 😁 vs YAML: "Hello 😁"), so that's fine 👍

[tamcy]

Just want to add that this is not just an emoji issue, but a more general one.

The yaml package uses string_scanner for string parsing, and it mostly utilizes the package's peekChar method, which returns the code unit of the string in a specific offset. We all know Dart uses UTF-16 in its internal representation, but the implementation of _isStandardCharacter() in scanner.dart obviously assumes a UTF-32 codepoint be supplied:

https://github.com/dart-lang/yaml/blob/e5984433a2803d5c67ed0abac5891a55040381ee/lib/src/scanner.dart#L1594-L1598

The last line checks for a range between 0x10000 and 0x10FFFF. If my observation is correct, this will never be true because char will never be larger than 0xFFFF. So, the actual issue is that the YAML parser doesn't handle surrogate pairs correctly for plain (unquoted) style. This does not only affects emojis. For instance, the following line is fine:

char: 兼

But the following would yield an "Unexpected character" exception:

char: 𠔥

Both are CJK characters, parsing would fail for the second case because 𠔥 (U+20525) requires two code units. This is what I have encountered. This requires YAML file author to remember to quote certain characters, or to quote all characters to play safe.

[kevmoo]

Maybe a PR to `string_scanner` is the place to start?

…

On Sat, Mar 23, 2024 at 8:46 PM Tamcy ***@***.***> wrote: Just want to add that this is not just an emoji issue, but a more general one. The yaml package uses string_scanner for string parsing, and it mostly utilizes the package's peekChar method, which returns the *code unit* of the string in a specific offset. We all know Dart uses UTF-16 in its internal representation, but the implementation of _isStandardCharacter() in scanner.dart obviously assumes a UTF-32 codepoint be supplied: https://github.com/dart-lang/yaml/blob/e5984433a2803d5c67ed0abac5891a55040381ee/lib/src/scanner.dart#L1594-L1598 The last line checks for a range between 0x10000 and 0x10FFFF. If my observation is correct, this will never be true because char will never be larger than 0xFFFF. So, the actual issue is that the YAML parser doesn't handle surrogate pairs correctly for plain (unquoted) style. This does not only affects emojis. For instance, the following line is fine: char: 兼 But the following would yield an "Unexpected character" exception: char: 𠔥 Both are CJK characters, parsing would fail for the second case because 𠔥 (U+20525) requires two code units. This is what I have encountered. This requires YAML file author to remember to quote certain characters, or to quote all characters to play safe. — Reply to this email directly, view it on GitHub <#1908>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEFCXMPRYK5MV5NHSPMWTYZZEC7AVCNFSM4HI44YE2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBRGY3DQMRWGM3Q> . You are receiving this because you commented.Message ID: <dart-lang/yaml/issues/53/2016682637 ***@***.***>

[tamcy]

(Edit: please see my comment below)

string_scanner seems fine, and the latest version (1.2.0) also added codepoint scanning which yaml can take advantage of to fix the problem I described. Perhaps after dart-lang/string_scanner#69 is accepted, which make .peekCodepoint() API compatible with .peekChar().

On the other hand, while there seems no harm to migrate the yaml package to use codepoint as the scanning unit (but it may cause performance issue), it also seems to me that changes like dart-lang/yaml#125 is already good enough or even inevitable. Here're my points:

1. It is unlikely to produce a malformed utf-16 string inadvertently, and in my opinion it should not be yaml package's responsibility to check against it.
2. I've run some simple test and found that the yaml package doesn't check against an invalid string if it is quoted. So I think the behavior should be the same for unquoted/plain strings. (I could be wrong here, I don't know the YAML spec good enough.)
3. It is still possible for the new .peekCodepoint() method to return a surrogate code point if the string is not formed correctly. And because of (2), yaml should still accept it as a part of the string, instead of throwing an invalid character exception. This means that it is still needed to allow the surrogate codepoint range in Scanner._isStandardCharacter(). The difference is that we should not remove the last check (char >= 0x10000 && char <= 0x10FFFF) if codepoint scanning is to be used.

[tamcy]

oh, I saw @kevmoo points to https://yaml.org/spec/1.2-old/spec.html#id2770814 in dart-lang/yaml#125. I've glanced through the document. It looks like the pull request wasn't accepted because the spec clearly excludes surrogate block from the allowed character range, so accepting \xD800-\xDFFF somehow violates the spec. But as said, currently the value passed into Scanner._isStandardCharacter() is actually a code unit (due to the fact that the peekChar() and readChar() methods in string_scanner actually return a code unit). In this case (or in order to adhere to the spec and fix the issue) bumping string_scanner to 1.2.0 (actually I mean something like 1.3.0, i.e. after my pull request is accepted) and make use of its new codepoint APIs seems the way to go. And maybe leave quoted string checking to the future.