Try to fix unicode layout issue

[skyline75489]

Summary of the Pull Request

This could be the first of a series PRs intended to fix unicode layout issue.

References

#3546

PR Checklist

• Closes #xxx
• CLA signed. If not, go over here and sign the CLA
• Tests added/passed
• Requires documentation to be updated
• I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Detailed Description of the Pull Request / Additional comments

Contrary to what @reli-msft emphasize in #3546, I found that zero length Unicode codepoint handling is the key here.

I searched a little and found https://gitlab.freedesktop.org/terminal-wg/specifications/issues/9 which refers https://github.com/ridiculousfish/widecharwidth. Then I was like, oh so there's bunch of unicode codepoint that's not included in WT currently.

Handling zero-width Unicode characters makes a lot of things weird. But here's what I got so far:

Would love to hear your guys' opinion on this.

Validation Steps Performed

[skyline75489]

Everything except Korean language is good now:

I couldn't make Hangul working because DWrite is treating them as they had single cell width. Maybe I missed something.

[reli-msft]

I think we need to do something at cell-allocation level, when we write a string into the buffer, we need to :

• Properly break the string into clusters (with consideration of combining/zwj/VS/...) and allocate a cell (1 or 2 column) for them;
• Categorize the clusters, decide:
  • Whether they are “adhensive”, i.e., whether they should be shaped together with the adjacent clusters under same category;
  • If their actual width is wider than the desired width, wwhat should we do (downscale/center/align left/align right)

And, when performing text layout, group adhensive clusters together and shape them together, then perform cluster-to-column alignment according to the cluster category.

Category flags:

enum ClusterCategory {
    ScriptCategoryMask   = 0xFF,
    // Non-adhensive scripts
    ScriptUnknown        = 0x00,
    ScriptIdeographs     = 0x01,
    ScriptEmoji          = 0x02,
    ScriptGeometric      = 0x03,
    ScriptNexusGeometric = 0x04, // Box drawing, Block Element, Powerline, etc.
    // Add more items when necessary
    Adhensive            = 0x80,
    // Adhensive scripts
    ScriptWestern        = 0x80,
    // Add more items when necessary

    // Alignment
    AlignCenter          = 0,
    AlignScaleDown       = 0x200,
    AlignLeft            = 0x400,
    AlignRight           = 0x600,
    AlignmentMask        = 0x600,

// Width
    DoubleWidth          = 0x400
}

[skyline75489]

@reli-msft Yeah I think you are right. Without cluster category, some code just doesn't really make sense. For example, in order to handle zero-width VS16 the text cell iterator is moving like this:

it += columnCount > 0 ? columnCount : 2;

If columnCount is zero, you need to move two cell ahead, because VS16 takes up space in string, just like ordinary CJK characters. That's just like "what the hell". The code can't explain itself, even with comments.

To thoroughly handle all kinds of Unicode thing, cluster category classification is a must.

[skyline75489]

This is definitely wrong since it's generated. Just showing the basic idea.

[reli-msft]

@skyline75489 I plan to have a meeting with Console people to find out a proper solution for this.
Since Windows Terminal do not care too much about compatibility, it is the time to do the right thing 😎.

[skyline75489]

@reli-msft You made my day, bro. Go get them.

[skyline75489]

@miniksa Come to think about it, maybe #3458 should be merged first. Right now, cluster handling code are scattered everywhere. #3458 at least give us a chance to handle cluster moving in one place inside RenderClusterIterator.

[DHowett-MSFT]

Since Windows Terminal do not care too much about compatibility, it is the time to do the right thing 😎.

Well, now, hold on. This is a good sentiment and all that, but still need to figure out how to deal with applications that read back the contents of the screen adn ahev their understanding of their codepoint to cell mapping violated. This isn't just a problem somebody can come in and solve overnight: Terminal still needs to act compatibly for legacy win32 console applications.

There's already #1472 tracking lighting up ZWJ/ZWNJ and any rendering and buffer concerns where N codepoints maps to M cells. This is work we know needs to be done, I just don't have the folkpower to staff it. 😄

Let it not be understated that we want to do the right thing here. It's just that we have to do the right thing in the context of 35 years worth of application support code and the fact that conhost (who also consumes this codepoint width detector, text buffer and text renderer) is the API server for all text-mode applications on every SKU of Windows everywhere.

[DHowett-MSFT]

To wit: WT can break UI paradigm compatibility and some level of application compatibility, but the code it shares with conhost must uphold the same compatibility guarantees as conhost does.

[skyline75489]

@DHowett-MSFT Easy. If I keep doing open source thing, the company I'm working at will soon be bankrupted and then I can work for MSFT 😎 . Everyone is happy.

Glad to know there's #1472. Gonna subscribe it right now.

[DHowett-MSFT]

LOL. 😁

[reli-msft]

I couldn't make Hangul working because DWrite is treating them as they had single cell width. Maybe I missed something.

Hangul is something more tricky: not all fonts support composite Hangul. I suggest we can isolate the Jamos. Or do NFKC-like composition...