unite our two control sequence automata

[dankamongmen]

We have two automata we traverse to handle control escapes -- the one handled by pump_control_read(), and the special key trie. as a result, we have two distinct control loops, and two distinct states. this last is horrible, as there's no well-defined map of state changes effected by one on the other.

unite these two automata. given that special keys require a dynamic state machine (it's prepared based on terminfo data), a wholly explicit controlflow-based automaton (like that used in pump_control_read()) won't work. we need a dataflow-based one, as used by our trie traversal.

so we'll need add some special node types to our trie, basically the same we have from the other automaton:

• accumulate a number
• accumulate a string
• accumulate a hex-encoded string, which is just a special case of either of the previous 2
  ...and that's it, i think? pretty easy. let's do it up right

[dankamongmen]

so right now a trienode just has an (optional) Σ-index (there is no index on terminating nodes). we will add a word for an aggregated numeric, string (pointer), an NCKEY, or function pointer to each node. since no sequences end in numerics or strings, this ought be sufficient for all nodes. the function pointer indicates an accepting state, and is invoked upon transition (since there are no valid prefixes, all accepting states are terminal states). we'll need a 4-way type; this can be two bits, but we'll likely just use a full int:

TRIE_NUMERIC
TRIE_STRING
TRIE_SPECIAL
TRIE_FXN

where TRIE_SPECIAL combined with NCKEY_INVALID (0) indicates a transitory node.

state-wise, we'll want an array of trienode pointers, equivalent in length to the longest path in the trie. we can calculate this as we build the trie, and allocate the array upon completion of trie construction. from this path plus the in-trie state, we can recreate any path without keeping the actual characters, meaning we can replay failed escapes as input without keeping external state.

Escape is handled distinctly, and always takes us to the STATE_ESCAPE node. this way we needn't have a Σ-index on terminal nodes, saving memory (each index is about 1K, with 128 ways and 8 bytes per pointer). Escape remains the initial node; we are not going to process standard UTF8 via the control trie. upon an escape while in-trie, we replay the path and return to ESC. upon a standard termination (non-0 TRIE_SPECIAL or any TRIE_FXN), we play in the special or execute on the fxn, and then move to NULL (do not move directly to STATE_ESCAPE).

we'll break apart the control-flow automaton, turning them into functions invoked on TRIE_FXN and (possibly manual) constructions. i don't think it's worth coming up with a little regexy DSL to do algorithmic construction -- there simply aren't enough TRIE_FXNs.

so for instance let's say we need recognize \e[num:num:numu, that would be

NULL Esc-> STATE_ESC [-> STATE_CSI 0-9-> TRIE_NUMERIC
0-9-> loop
:-> TRIE_SPECIAL[0] 0-9-> TRIE_NUMERIC
0-9->loop
:-> TRIE_SPECIAL[0] 0-9-> TRIE_NUMERIC
0-9->loop
u-> TRIE_FXN

where that TRIE_FXN takes the three aggregated numerics, evaluates them, and calls the relevant function. we then move back to NULL. any Escape along the way moves us to STATE_ESC following a replay along the path. any invalid follow moves us to NULL following a replay along the path.

that ought work.

[dankamongmen]

also, trienodes ought hold an ncinput, not their own weird assemblage of ncinput.

[dankamongmen]

ok, i've gotta stop tonight, but very good progress. i've gone ahead and done a DSL, with \N as a numeric accumulator, \S as a string accumulator, and literals otherwise. we'll want \H as the hash, and we might need a Kleene closure for drainage purposes. the constaints for numeric transitions are too tight right now--i've had to comment out the special keys for now, but we'll need add them back. that same principle would apply to any Kleene.

i've converted about half of the CF automaton over to DF -- mouse, kitty, cursor location, and geometry.

when using the kitty protocol, we probably ought not try to match old-school alts/ctrls.

so this will be done soon. dankamongmen/unifytries branch.

[dankamongmen]

well, we have verified that unknown sequences even when interleaved with the initial query responses now get replayed as input, which is good! exactly what we expected.

[wez]

[dankamongmen]

5fccc295-1b29-4916-a696-da948c9d80b5.mp4

[dankamongmen]

wezterm now starts without any bleed, 'twas a bug in our Kleene reduction

[dankamongmen]

this also fixed the XTerm bleed. i no longer see bleeding with any terminal. =]

[dankamongmen]

btw @wez while i've got your attention, i added some info to wez/wezterm#1063

[dankamongmen]

mlterm bleeds with \e[?1;3;0S

[dankamongmen]

mlterm is handled

[dankamongmen]

st is fine

[dankamongmen]

VTE bleed: \e[?1;1S\e[?2;1S\e[?1;1S

[dankamongmen]

no more bleed for VTE

[dankamongmen]

hyper is fine

[dankamongmen]

contour is fine

[dankamongmen]

so we only have one more problem that i see: in all terminals, the intro banner is printed over, unless we're at the bottom. presumably a missing cursor location lookup in init_banner()?

[dankamongmen]

hrmmm. when redirected, and we then cat the redirected output, the same thing happens with 2.4.3 (which does not exhibit this behavior when not redirected). maybe this is related to #2114?

[dankamongmen]

got the last issue resolved. =]

[dankamongmen]

nope, one more problem: we're getting a stray 'u' in notcurses-demo on kitty only.

[dankamongmen]

ok, if i throw a sleep() in before the render in notcurses-info, and press ctrl-c, the u is printed following the ctrl-c:

[dankamongmen]

hrmmmm it seems to be KKEYBOARD_POP...

[dankamongmen]

oh this is an old version of kitty (0.19.3). yeah, looks like popping the keyboard protocol results in a u in 0.19.3 (it looks like the keyboard protocol was introduced in 0.20.0):

[schwarzgerat](0) $ echo -e '\e[<u'
u
[schwarzgerat](0) $ 

[dankamongmen]

we should probably just refrain from emitting a KKEYBOARD_POP unless we got a response to \e[?u. We're currently sending it unconditionally.

[dankamongmen]

got it, fixed

[BarryHuffman]

Slightly off topic but as a non-dev I enjoyed reading your thoughts on sprixels and your proposals on STEGP.

With RHEL/Canonical's focusing more on desktop, and neglecting console interfaces (removal of the TTY/Console scrollback feature being a major example), issues in KMS/DRM, undocumented lack of legacy VT options or POSIX, ect.

It's great that Notcurses is pushing the envelope, and exposing the cracks in "modern" vterminals/consoles/libraries.