Support EXR/WebP using direct transmission

[dankamongmen]

Is your feature request related to a problem? Please describe.
Kitty can display at least JPEG, EXR, and WebP when using indirect transmission media. When providing the data directly, however (t=d), the only documented format is PNG (f=100). I'd like to explore EXR and WebP, due to their superior characteristics viz PNG on certain images, and I intend to use direct transmission (generally necessary for remote clients, AFAICT). This is related to dankamongmen/notcurses#1821 and dankamongmen/notcurses#1695, where I'm seeing nice performance improvements over RGBA.

Describe the solution you'd like
I'd like f=101 for EXR and f=102 for WebP, or whatever. I do not think it sensible to do arbitrary container detection along this path; if you want to do that, I'd suggest f=420 or whatever, so that clients that know their format ahead of time (as I will) can specify it.

Describe alternatives you've considered
Always encoding to PNG, which is fine except these might be better (and you already seem to support them), and transporting RGBA directly, which is (usually) slower due to the (usually) greater bandwidth.

Additional context
aside: I'd still really, really like to get away from base64 encoding, but kinda understand why you don't want to completely disable the lexer state machine for the duration of reading graphics. Given that failing to complete a graphic can already lock Kitty up, requiring a reset, is it really necessary to watch for other types of escapes? It's unfortunate.

[kovidgoyal]

On Thu, Jun 24, 2021 at 12:08:48AM -0700, Nick Black wrote: **Is your feature request related to a problem? Please describe.** Kitty can display at least JPEG, EXR, and WebP when using indirect transmission media. When providing the data directly, however (`t=d`), the only documented format is PNG (`f=100`). I'd like to explore EXR and WebP, due to their superior characteristics viz PNG on certain images, and I intend to use direct transmission (generally necessary for remote clients, AFAICT). This is related to dankamongmen/notcurses#1821 and dankamongmen/notcurses#1695, where I'm seeing nice performance improvements over RGBA.

kitty can display only RGB(A) and PNG data. All other formats are converted by the icat kitten under the hood, by invoking ImageMagick. kitty itself links *only* to libpng. Adding another image format would require adding another decoder library dependency. So while I am not totally opposed to it, it would require a pretty high bar for qualification.

**Describe the solution you'd like** I'd like `f=101` for EXR and `f=102` for WebP, or whatever. I do not think it sensible to do arbitrary container detection along this path; if you want to do that, I'd suggest `f=420` or whatever, so that clients that know their format ahead of time (as I will) can specify it.

I explicitly dont want to do arbitrary container detection. The graphics protocol must have a limited set of well defined graphics data interchange formats. Terminals, unlike browsers, are not going to be forever adding support for new image formats. I value stability over optimal bandwidth.

aside: I'd still really, really like to get away from base64 encoding, but kinda understand why you don't want to completely disable the lexer state machine for the duration of reading graphics. Given that failing to complete a graphic can already lock Kitty up, requiring a reset, is it really necessary to watch for other types of escapes? It's unfortunate.

You were the one that asked for a simple way to abort any in-flight escape code. I added NUL as that way to kitty. So any alternative encoding scheme would have to avoid at least the bytes NUL, ST (ESC + \ as well as its C1 encoding) and BEL. That's four bytes you could not have in any alternative encoding scheme already and that's just keeping in mind kitty's parser. you would need your scheme to work with many other terminal, multiplexers, etc. Instead, I strongly suggest you make use of out-of-band transmission. The common case for viewing high resolution graphics is not viewing them over a network connection, but locally, and out of band transmission completely obviates the need for any encoding since it allows direct transmission of binary data.

[dankamongmen]

thanks for the detailed feedback, got it!

[dankamongmen]

i think i will look into OOB transmission after all, especially after seeing the performance numbers in dankamongmen/notcurses#1695. they're simply too tantilizing!

[kovidgoyal]

Indeed, I view in band transmission of image data as very much a last reesort, network only fallback. The icat kitten for instance uses files for OOB which makes sense for it since imagemagick outputs files.