MacVim does not display Tibetan Text properly #995
Labels
Renderer
Text rendering issues, including CoreText renderer
Milestone
Comments
|
@madmansnest Thank you for an issue, It's true, that some font parameters are not loaded from font correctly, but not many people was interested in it earlier. You can try to modify |
|
Could you share with us a bit more information, such as some texts and the font you use (it's home page is preferable and tell if it's free-to-use (it'll be much harder to test the same over a paid one) |
|
Text Example: |
|
MacVim actually doesn’t use NSTextView by default and uses CoreText renderer to get more flexibility in how it renders text. I will need to take a look but this is likely similar to other font issues we have had. |
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 21, 2023
MacVim previously didn't really render composing characters with multiple glyphs correctly. For simple ones like 'â' it would work fine because Core Text just generates one glyph for it, but for more complicated ones like 'x⃗' the renderer didn't properly query the advance of the glyphs to put them at the correct position. Refactor the logic to keep track of the current cell/glyphs and make sure to track the advances fo the glyphs so the composing chars' glyphs would be laid out properly on top of the cell. We need to be careful with the tracking, because in Vim we force the text rendering to be monospaced, and so we maintain the tracking within a single grapheme (which can consist of multiple glyphs), but when we move to the next grapheme we reset the position so we can get proper monospaced rendering even for non-monospaced texts like CJK or stylized texts. Fix macvim-dev#995 Fix macvim-dev#1172
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 21, 2023
This fixes the issue where particularly tall characters will get clipped at the row boundary. This happens because even though a font describes the line height with font metrics, individual glyphs do not have to respect them, and we can see with emoji rendering sometimes they can poke upwards past the line height. Also, it's trivially easy to construct composing characters that become really tall, e.g. "x゙̂⃗", or Tibetan scripts like "ཧཱུྃ". To fix this, we do the following: 1. Remove the explicit clipping call at rendering. 2. Fix partial redraw to not lead to clipping / corruption. This is quite tricky, because let's say we have a character that is tall enough to touch other lines, if those lines are redraw but not the line with the tall char, the redraw will paint over the parts of the glyphs poking through. Alternatively if we redraw the line with the tall chars we also need to expand the redraw region to make sure other lines get repainted as well. To fix this properly, we should do a proper glyph calculation when we receive the draw command before we issue before we call `setNeedsDisplayInRect`, but since right now we only extract glyph info later (during drawRect call), it's too late. We just do the hacky solution by storing a variable `redrawExpandRows` that tracks how many lines to expand for all lines. It's a little hacky since this will affect all lines permanently regardless if they have tall characters or not. The proper fix may come later as an optimization (or when we do hardware rendering via Metal). 3. Re-order the rendering so we have a two pass system, where we first draw the background fill color for all rows, then the text. This helps prevent things like Vim's window split or cursorline from obscuring the text. 4. Add a preference to turn on clipping (old behavior). This helps prevent odd issues with really tall texts (e.g. Zalgo text) making it hard to see what's going on. The preference `MMRendererClipToRow` is not exposed in UI for now as it's a relatively niche. It will be exposed later when we have a dedicated render tab in settings. Note that a lot of these characters only show their full height by doing `set maxcombine=8` because the default (2) is quite low. Fix macvim-dev#456 Part of the fix for macvim-dev#995
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 21, 2023
MacVim previously didn't really render composing characters with multiple glyphs correctly. For simple ones like 'â' it would work fine because Core Text just generates one glyph for it, but for more complicated ones like 'x⃗' the renderer didn't properly query the advance of the glyphs to put them at the correct position. Refactor the logic to keep track of the current cell/glyphs and make sure to track the advances fo the glyphs so the composing chars' glyphs would be laid out properly on top of the cell. We need to be careful with the tracking, because in Vim we force the text rendering to be monospaced, and so we maintain the tracking within a single grapheme (which can consist of multiple glyphs), but when we move to the next grapheme we reset the position so we can get proper monospaced rendering even for non-monospaced texts like CJK or stylized texts. Fix macvim-dev#995 Fix macvim-dev#1172
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 21, 2023
This fixes the issue where particularly tall characters will get clipped at the row boundary. This happens because even though a font describes the line height with font metrics, individual glyphs do not have to respect them, and we can see with emoji rendering sometimes they can poke upwards past the line height. Also, it's trivially easy to construct composing characters that become really tall, e.g. "x゙̂⃗", or Tibetan scripts like "ཧཱུྃ". To fix this, we do the following: 1. Remove the explicit clipping call at rendering. 2. Fix partial redraw to not lead to clipping / corruption. This is quite tricky, because let's say we have a character that is tall enough to touch other lines, if those lines are redraw but not the line with the tall char, the redraw will paint over the parts of the glyphs poking through. Alternatively if we redraw the line with the tall chars we also need to expand the redraw region to make sure other lines get repainted as well. To fix this properly, we should do a proper glyph calculation when we receive the draw command before we issue before we call `setNeedsDisplayInRect`, but since right now we only extract glyph info later (during drawRect call), it's too late. We just do the hacky solution by storing a variable `redrawExpandRows` that tracks how many lines to expand for all lines. It's a little hacky since this will affect all lines permanently regardless if they have tall characters or not. The proper fix may come later as an optimization (or when we do hardware rendering via Metal). 3. Re-order the rendering so we have a two pass system, where we first draw the background fill color for all rows, then the text. This helps prevent things like Vim's window split or cursorline from obscuring the text. 4. Add a preference to turn on clipping (old behavior). This helps prevent odd issues with really tall texts (e.g. Zalgo text) making it hard to see what's going on. The preference `MMRendererClipToRow` is not exposed in UI for now as it's a relatively niche. It will be exposed later when we have a dedicated render tab in settings. Note that a lot of these characters only show their full height by doing `set maxcombine=8` because the default (2) is quite low. Part of the fix for macvim-dev#995
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 29, 2023
MacVim previously didn't really render composing characters with multiple glyphs correctly. For simple ones like 'â' it would work fine because Core Text just generates one glyph for it, but for more complicated ones like 'x⃗' the renderer didn't properly query the advance of the glyphs to put them at the correct position. Refactor the logic to keep track of the current cell/glyphs and make sure to track the advances fo the glyphs so the composing chars' glyphs would be laid out properly on top of the cell. We need to be careful with the tracking, because in Vim we force the text rendering to be monospaced, and so we maintain the tracking within a single grapheme (which can consist of multiple glyphs), but when we move to the next grapheme we reset the position so we can get proper monospaced rendering even for non-monospaced texts like CJK or stylized texts. Fix macvim-dev#995 Fix macvim-dev#1172
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 29, 2023
This fixes the issue where particularly tall characters will get clipped at the row boundary. This happens because even though a font describes the line height with font metrics, individual glyphs do not have to respect them, and we can see with emoji rendering sometimes they can poke upwards past the line height. Also, it's trivially easy to construct composing characters that become really tall, e.g. "x゙̂⃗", or Tibetan scripts like "ཧཱུྃ". To fix this, we do the following: 1. Remove the explicit clipping call at rendering. 2. Fix partial redraw to not lead to clipping / corruption. This is quite tricky, because let's say we have a character that is tall enough to touch other lines, if those lines are redraw but not the line with the tall char, the redraw will paint over the parts of the glyphs poking through. Alternatively if we redraw the line with the tall chars we also need to expand the redraw region to make sure other lines get repainted as well. To fix this properly, we should do a proper glyph calculation when we receive the draw command before we issue before we call `setNeedsDisplayInRect`, but since right now we only extract glyph info later (during drawRect call), it's too late. We just do the hacky solution by storing a variable `redrawExpandRows` that tracks how many lines to expand for all lines. It's a little hacky since this will affect all lines permanently regardless if they have tall characters or not. The proper fix may come later as an optimization (or when we do hardware rendering via Metal). 3. Re-order the rendering so we have a two pass system, where we first draw the background fill color for all rows, then the text. This helps prevent things like Vim's window split or cursorline from obscuring the text. 4. Add a preference to turn on clipping (old behavior). This helps prevent odd issues with really tall texts (e.g. Zalgo text) making it hard to see what's going on. The preference `MMRendererClipToRow` is not exposed in UI for now as it's a relatively niche. It will be exposed later when we have a dedicated render tab in settings. Note that a lot of these characters only show their full height by doing `set maxcombine=8` because the default (2) is quite low. Part of the fix for macvim-dev#995
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 30, 2023
MacVim previously didn't really render composing characters with multiple glyphs correctly. For simple ones like 'â' it would work fine because Core Text just generates one glyph for it, but for more complicated ones like 'x⃗' the renderer didn't properly query the advance of the glyphs to put them at the correct position. Refactor the logic to keep track of the current cell/glyphs and make sure to track the advances fo the glyphs so the composing chars' glyphs would be laid out properly on top of the cell. We need to be careful with the tracking, because in Vim we force the text rendering to be monospaced, and so we maintain the tracking within a single grapheme (which can consist of multiple glyphs), but when we move to the next grapheme we reset the position so we can get proper monospaced rendering even for non-monospaced texts like CJK or stylized texts. Fix macvim-dev#995 Fix macvim-dev#1172
ychin
added a commit
to ychin/macvim
that referenced
this issue
Jan 30, 2023
This fixes the issue where particularly tall characters will get clipped at the row boundary. This happens because even though a font describes the line height with font metrics, individual glyphs do not have to respect them, and we can see with emoji rendering sometimes they can poke upwards past the line height. Also, it's trivially easy to construct composing characters that become really tall, e.g. "x゙̂⃗", or Tibetan scripts like "ཧཱུྃ". To fix this, we do the following: 1. Remove the explicit clipping call at rendering. 2. Fix partial redraw to not lead to clipping / corruption. This is quite tricky, because let's say we have a character that is tall enough to touch other lines, if those lines are redraw but not the line with the tall char, the redraw will paint over the parts of the glyphs poking through. Alternatively if we redraw the line with the tall chars we also need to expand the redraw region to make sure other lines get repainted as well. To fix this properly, we should do a proper glyph calculation when we receive the draw command before we issue before we call `setNeedsDisplayInRect`, but since right now we only extract glyph info later (during drawRect call), it's too late. We just do the hacky solution by storing a variable `redrawExpandRows` that tracks how many lines to expand for all lines. It's a little hacky since this will affect all lines permanently regardless if they have tall characters or not. The proper fix may come later as an optimization (or when we do hardware rendering via Metal). 3. Re-order the rendering so we have a two pass system, where we first draw the background fill color for all rows, then the text. This helps prevent things like Vim's window split or cursorline from obscuring the text. 4. Add a preference to turn on clipping (old behavior). This helps prevent odd issues with really tall texts (e.g. Zalgo text) making it hard to see what's going on. The preference `MMRendererClipToRow` is not exposed in UI for now as it's a relatively niche. It will be exposed later when we have a dedicated render tab in settings. Note that a lot of these characters only show their full height by doing `set maxcombine=8` because the default (2) is quite low. Part of the fix for macvim-dev#995
ychin
added a commit
that referenced
this issue
Feb 7, 2023
Updated to Vim 9.0.1276 Features ==================== Dictionary lookup -------------------- You can now use Force Touch or Cmd-Ctrl-D to look up definitions of word under the cursor (or selected text in visual mode). This will also preview URLs, and support data types such as phone numbers and addresses. #1312 #1313 This feature can also be invoked programmatically from VimScript (see `:h macvim-lookup`). #1315 Tool bar / Touch Bar / menu icons -------------------- You can now use SF Symbols for Tool bar and Touch Bar icons, including using different symbol styles such as "palette" or "multicolor". Menu items can now also use the `icon=` syntax to specify icons as well. See `:help macvim-toolbar-icon` for details. #1329 The default tool bar also has updated icons to look similar to SF Symbols used by newer macOS versions. #1214 by @sfsam Window management actions -------------------- There are new `macaction`'s for managing the MacVim window. The new `zoomLeft`/`zoomRight` actions allow you to pin the window to the left/right of the screen, and there are also new actions for interacting with Stage Manager (requires macOS 13+). See `:h macvim-actions` for details. #1330 Pre-release updates / Sparkle 2 -------------------- MacVim now supports pre-release software builds. It's sometimes hard for us to release frequent updates due to the desire to pick a stable upstream Vim version, needing to test the release on multiple OS versions, making sure there aren't half-complete or buggy features, and other reasons. This new feature now allows us to push pre-release beta builds out in a more frequent fashion, which could be useful if there are particular features or fixes that you would like to try out before the next official release. Pre-release builds will be released depending on bug fixes and features instead of a fixed cadence. Do note that these pre-release builds may not be as well-validated and may have half-baked features. If you are using the built-in auto-updater to update MacVim, you can turn this on by going to Advanced settings pane, and enable "Enable pre-release software updates". This feature is only available for macOS 10.13 or above. The auto-updater has also been updated from Sparkle 1.27.1 to 2.3.0 for 10.13+ builds. Legacy (10.9-10.12) builds are still using Sparkle 1. See #1332. New Vim features -------------------- New `smoothscroll` option allows you to scroll through a long wrapped line (using Ctrl-E or mouse wheel) without immediately jumping to the next line. (v9.0.0640) `splitscroll` option has been renamed `splitkeep`, with more flexibility than before. (v9.0.0647) Sound playback on macOS is now supported. You can use `has('sound')` to check. See `help sound` for details. (v9.0.0694) Terminals now support `:confirm` for `:q`, etc, which also means MacVim's Cmd-W will work properly for terminal windows. (v9.0.0710) Virtual text had numerous bugs fixed. General ==================== Legacy build for 10.9 - 10.12 -------------------- Per a previous announcement (#1271), the default MacVim binary will now require macOS 10.13 or above. Users of macOS 10.9 - 10.12 can use a separate "legacy" build which will still be supported. The legacy binary will still have the latest versions of Vim and be supported, but may not have all the latest features (e.g. pre-release builds). If you are using the auto-updater (Sparkle) to update MacVim, it should "just work" and find the best version for you. If you are downloading MacVim from the website, there is also a link to download the legacy version marked for 10.9+ as well. If you download the normal binary marked for 10.13+ from the website, it won't work on these older macOS versions. See #1331. Fixes ==================== CoreText Renderer clipping and rendering bugs -------------------- Unicode characters with multiple composing characters (e.g. "x⃗") will now render correctly. #1172 Texts (e.g. Tibetan, Zalgo texts) that are taller than the line height will no longer be clipped inappropriately. You can use a new setting `MMRendererClipToRow` to re-enable clipping if the tall texts are distracting. #995 / #1356 Tab crash -------------------- Fixed a crash when opening new tabs that seems to only occur in macOS 13 Ventura. #1333 Other bugs -------------------- - Fixed non-native full screen not working well with the notch on newer MacBook's when set to not show menu bar. You can also use `MMNonNativeFullScreenSafeAreaBehavior` to force MacVim to use the notch area as well if you don't mind some content being obscured. Note that the previous release also claimed it fixed this, but because the binary was built against an old macOS SDK (Big Sur), the fix did not work in the binary release. #1261 - Allow "Open untitled window: never" and "After last window closes: Quit MacVim" to be set together again. Added safeguards to make sure doing so won't immediately close the app. #1338 - Edit.Cut / Copy menu items will now be properly disabled when there isn't selected text. #1308 - Fixed potential `:emenu` crash when the menu is associated with an action in a non-valid mode. #1305 - Fixed bug where just bringing up the right-click (or the MacVim→Services) menu would somehow copy the selected texts to the system clipboard. #1300 - Fixed a Japanese input method bug where using left/right arrow to move to a different section of the input text would previously result in the candidate list not showing up at the correct position. #1312 - Fix non-CoreText renderer not handling text styles like strikethrough correctly (note: this renderer has been deprecated for a while and you should not use it). #1296 - This release uses an older sh/bash syntax file because the latest one in Vim has a bug. #1358 Misc ==================== New settings: - "No drop shadows" (Appearance). #1301 - "Treat Ctrl-click as right-click" (Input) (#1326). This was previously configurable via command-line, but now also possible in the settings pane under the new "Input" category. "About MacVim" now reports the version number in a clearer way with clearly specified release number vs Vim version. Known Issues ==================== Printing -------------------- Printing using File→Print or `:hardcopy` is currently not working under macOS 13 Ventura due to its removal of PostScript support in the Preview app. This will be fixed in a later release. See the issue for workarounds. #1347 Scripting ==================== - Scripting languages versions: - Perl is now built against 5.30, up from 5.18. - Ruby is now built against 3.2, up from 3.1. Compatibility ==================== Requires macOS 10.9 or above. (10.9 - 10.12 requires downloading a separate legacy build) Script interfaces have compatibility with these versions: - Lua 5.4 - Perl 5.30 - Python2 2.7 - Python3 3.10 - Ruby 3.2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Even though in latest OS X versions there has been no problems with Tibetan text display, and even though MacVim uses the standard NSTextView, it still fails to display Tibetan text properly, just like Terminal.app used to many OS X versions ago.
To Reproduce: Open MacVim and input some relatively complex Tibetan text like ཧཱུྃ། or བསྒྲིགས་པ་. In order to see what it should look like, open TextEdit and input the same text.
This is current MacVim, unsatisfactory:
This is Terminal.app in OS X Mojave, bad kerning, but satisfactory:
This is what is should like, ideal:
The text was updated successfully, but these errors were encountered: