Skip to content

Commit

Permalink
A minor ConPTY refactoring: Goodbye VtEngine Edition (#17510)
Browse files Browse the repository at this point in the history
The idea is that we can translate Console API calls directly to VT at
least as well as the current VtEngine setup can. For instance, a call
to `SetConsoleCursorPosition` clearly translates directly to a `CUP`
escape sequence. Effectively, instead of translating output
asynchronously in the renderer thread, we'll do it synchronously
right during the Console API call.

Most importantly, the this means that any VT output that an
application generates will now be given to the terminal unmodified.

Aside from reducing our project's complexity quite a bit and opening
the path towards various interesting work like sixels, Device Control
Strings, buffer snapshotting, synchronized updates, and more, it also
improves performance for mixed text output like enwik8.txt in conhost
to 1.3-2x and in Windows Terminal via ConPTY to roughly 20x.

This adds support for overlapped IO, because now that output cannot
be "skipped" anymore (VtEngine worked like a renderer after all)
it's become crucial to block conhost as little as possible.

⚠️ Intentionally unresolved changes/quirks:
* To force a delayed EOL wrap to wrap, `WriteCharsLegacy` emits a
  `\r\n` if necessary. This breaks text reflow on window resize.
  We cannot emit ` \r` the way readline does it, because this would
  overwrite the first column in the next row with a whitespace.
  The alternative is to read back the affected cell from the buffer
  and emit that character and its attributes followed by a `\r`.
  I chose to not do that, because buffer read-back is lossy (= UCS2).
  Unless the window is resized, the difference is unnoticeable
  and historically, conhost had no support for buffer reflow anyway.
* If `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is set while
  `DISABLE_NEWLINE_AUTO_RETURN` is reset, we'll blindly replace all
  LF with CRLF. This may hypothetically break DCS sequences, but it's
  the only way to do this without parsing the given VT string and
  thus the only way we can achieve passthrough mode in the future.
* `ENABLE_WRAP_AT_EOL_OUTPUT` is translated to `DECAWM`.
  Between Windows XP and Windows 11 21H2, `ENABLE_WRAP_AT_EOL_OUTPUT`
  being reset would cause the cursor position to reset to wherever
  a write started, _if_ the write, including expanded control chars,
  was less than 100 characters long. If it was longer than that,
  the cursor position would end up in an effectively random position.
  After lengthy research I believe that this is a bug introduced in
  Windows XP and that the original intention was for this mode to be
  equivalent to `DECAWM`. This is compounded by MSDN's description
  (emphasis mine):
  > If this mode is disabled, the **last character** in the row is
  > overwritten with any subsequent characters.

⚠️ Unresolved issues/quirks:
* Focus/Unfocus events are injected into the output stream without
  checking whether the VT output is currently in a ground state.
  This may break whatever VT sequence is currently ongoing.
  This is an existing issue.
* `VtIo::Writer::WriteInfos` should properly verify the width of
  each individual character.
* Using `SetConsoleActiveScreenBuffer` destroys surrogate pairs
  and extended (VT) attributes. It could be translated to VT pages
  in the long term.
* Similarly, `ScrollConsoleScreenBuffer` results in the same and
  could be translated to `DECCRA` and `DECFRA` in the near term.
  This is important because otherwise `vim` output may loose
  its extended attributes during scrolling.
* Reflowing a long line until it wraps results in the cooked read
  prompt to be misaligned vertically.
* `SCREEN_INFORMATION::s_RemoveScreenBuffer` should trigger a
  buffer switch similar to `SetConsoleActiveScreenBuffer`.
* Translation of `COMMON_LVB_GRID_HORIZONTAL` to `SGR 53` was dropped
  and may be reintroduced alongside `UNDERSCORE` = `SGR 4`.
* Move the `OSC 0 ; P t BEL` sequence to `WriteWindowTitle`
  and swap the `BEL` with the `ST` (`ESC \`).
* PowerShell on Windows 10 ships with PSReadLine 2.0.0-beta2
  which emits SGR 37/40 instead of 39/49. This results in black
  spaces when typing and there's no good way to fix that.
* A test is missing that ensures that `FillConsoleOutputCharacterW`
  results in a `CSI n J` during the PowerShell shim.
* A test is missing that ensures that `PtySignal::ClearBuffer`
  does not result in any VT being generated.

Closes #262
Closes #1173
Closes #3016
Closes #4129
Closes #5228
Closes #8698
Closes #12336
Closes #15014
Closes #15888
Closes #16461
Closes #16911
Closes #17151
Closes #17313
  • Loading branch information
lhecker authored Aug 1, 2024
1 parent a858297 commit 450eec4
Show file tree
Hide file tree
Showing 125 changed files with 1,974 additions and 17,468 deletions.
3 changes: 2 additions & 1 deletion .github/actions/spelling/allow/apis.txt
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ GETTEXTLENGTH
Hashtable
HIGHCONTRASTON
HIGHCONTRASTW
hinternet
HIGHQUALITYSCALE
hinternet
HINTERNET
hotkeys
href
Expand Down Expand Up @@ -155,6 +155,7 @@ NOTIFYBYPOS
NOTIFYICON
NOTIFYICONDATA
ntprivapi
NTSYSCALLAPI
numr
oaidl
ocidl
Expand Down
7 changes: 0 additions & 7 deletions .github/actions/spelling/expect/alphabet.txt
Original file line number Diff line number Diff line change
@@ -1,20 +1,15 @@
AAAAA
AAAAAAAAAAAAA
AAAAAABBBBBBCCC
AAAAABBBBBBCCC
abcd
abcd
ABCDEFGHIJ
abcdefghijk
ABCDEFGHIJKLMNO
abcdefghijklmnop
ABCDEFGHIJKLMNOPQRS
ABCDEFGHIJKLMNOPQRST
ABCG
ABE
abf
BBBBB
BBBBBBBB
BBBBBCCC
BBBBCCCCC
BBGGRR
Expand All @@ -29,10 +24,8 @@ QQQQQQQQQQABCDEFGHIJKLMNOPQRSTQQQQQQQQQ
QQQQQQQQQQABCDEFGHIJKLMNOPQRSTQQQQQQQQQQ
QQQQQQQQQQABCDEFGHIJPQRST
QQQQQQQQQQABCDEFGHIJPQRSTQQQQQQQQQQ
qrstuvwxyz
qwerty
qwertyuiopasdfg
YYYYYYYDDDDDDDDDDD
ZAAZZ
ZABBZ
ZBAZZ
Expand Down
25 changes: 7 additions & 18 deletions .github/actions/spelling/expect/expect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ ADDALIAS
ADDREF
ADDSTRING
ADDTOOL
AFew
AFill
AFX
AHelper
Expand Down Expand Up @@ -66,7 +65,6 @@ ARRAYSIZE
ARROWKEYS
asan
ASBSET
asdfghjkl
ASetting
ASingle
ASYNCDONTCARE
Expand Down Expand Up @@ -125,6 +123,7 @@ BKCOLOR
BKGND
Bksp
Blt
blu
BLUESCROLL
bmi
BODGY
Expand All @@ -145,7 +144,6 @@ buffersize
buflen
buildtransitive
buildsystems
burriter
BValue
bytebuffer
cac
Expand Down Expand Up @@ -210,7 +208,6 @@ cmw
CNL
cnn
Codeflow
codenav
codepages
codepath
coinit
Expand Down Expand Up @@ -362,6 +359,7 @@ DBGFONTS
DBGOUTPUT
dbh
dblclk
Dcd
DColor
DCOLORVALUE
dcommon
Expand All @@ -379,7 +377,7 @@ DECALN
DECANM
DECARM
DECAUPSS
DECAWM
decawm
DECBI
DECBKM
DECCARA
Expand Down Expand Up @@ -421,6 +419,7 @@ DECPCCM
DECPCTERM
DECPS
DECRARA
decrc
DECRC
DECREQTPARM
DECRLM
Expand All @@ -436,6 +435,7 @@ DECRSPS
decrst
DECSACE
DECSASD
decsc
DECSC
DECSCA
DECSCNM
Expand Down Expand Up @@ -475,7 +475,6 @@ DEFPUSHBUTTON
defterm
DELAYLOAD
DELETEONRELEASE
Delt
depersist
deprioritized
deserializers
Expand Down Expand Up @@ -556,7 +555,6 @@ Efast
efghijklmn
EHsc
EINS
EJO
ELEMENTNOTAVAILABLE
EMPTYBOX
enabledelayedexpansion
Expand Down Expand Up @@ -626,7 +624,6 @@ FINDDOWN
FINDREGEX
FINDSTRINGEXACT
FINDUP
FIter
FITZPATRICK
FIXEDFILEINFO
Flg
Expand Down Expand Up @@ -723,6 +720,7 @@ GETWHEELSCROLLLINES
Gfun
gfx
GGI
GHgh
GHIJK
GHIJKL
gitcheckin
Expand Down Expand Up @@ -948,7 +946,6 @@ LCONTROL
LCTRL
lcx
LEFTALIGN
libpopcnt
libsancov
libtickit
licate
Expand Down Expand Up @@ -1046,7 +1043,6 @@ MAPBITMAP
MAPVIRTUALKEY
MAPVK
MAXDIMENSTRING
maxing
MAXSHORT
maxval
maxversiontested
Expand Down Expand Up @@ -1506,7 +1502,6 @@ REGSTR
RELBINPATH
remoting
renamer
renderengine
rendersize
reparented
reparenting
Expand Down Expand Up @@ -1847,7 +1842,6 @@ Trd
TREX
triaged
triaging
Tribool
TRIMZEROHEADINGS
trx
tsa
Expand Down Expand Up @@ -1941,7 +1935,6 @@ uxtheme
Vanara
vararg
vclib
vcprintf
vcxitems
vectorize
VERCTRL
Expand Down Expand Up @@ -1985,7 +1978,6 @@ vtio
vtmode
vtpipeterm
vtpt
vtrenderer
VTRGB
VTRGBTo
vtseq
Expand All @@ -2007,6 +1999,7 @@ wcswidth
wddm
wddmcon
WDDMCONSOLECONTEXT
WDK
wdm
webpage
websites
Expand Down Expand Up @@ -2074,7 +2067,6 @@ Winperf
WInplace
winres
winrt
wintelnet
winternl
winuser
winuserp
Expand Down Expand Up @@ -2175,7 +2167,6 @@ XTWINOPS
xunit
xutr
XVIRTUALSCREEN
XWalk
yact
YCast
YCENTER
Expand All @@ -2184,14 +2175,12 @@ YLimit
YPan
YSubstantial
YVIRTUALSCREEN
YWalk
Zab
zabcd
Zabcdefghijklmn
Zabcdefghijklmnopqrstuvwxyz
ZCmd
ZCtrl
ZWJs
zxcvbnm
ZYXWVU
ZYXWVUTd
65 changes: 0 additions & 65 deletions NOTICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,71 +84,6 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
```

## kimwalisch/libpopcnt

**Source**: [https://github.com/kimwalisch/libpopcnt](https://github.com/kimwalisch/libpopcnt)

### License

```
BSD 2-Clause License
Copyright (c) 2016 - 2019, Kim Walisch
Copyright (c) 2016 - 2019, Wojciech Muła
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
```

## dynamic_bitset

**Source**: [https://github.com/pinam45/dynamic_bitset](https://github.com/pinam45/dynamic_bitset)

### License

```
MIT License
Copyright (c) 2019 Maxime Pinard
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```

## \{fmt\}

**Source**: [https://github.com/fmtlib/fmt](https://github.com/fmtlib/fmt)
Expand Down
Loading

0 comments on commit 450eec4

Please sign in to comment.