[RFC] Custom window shader in experimental OpenGL backend

[tryone144]

Implementation of an advanced interface for custom window shaders in the new experimental OpenGL backend (see #386). This will offer similar functionality to the existing glx-fshader-win option in the legacy backend but should be easier to use and cover more use-cases.

WARNING: This is currently just a Proof-of-Concept to gather some feedback on the interface.
This has known bugs and will likely crash with an improper config as some checks are still missing.

Implementation details

The implementation is up for debate. Comments and ideas for improvements welcome!

Configuration

Custom (foreground) window shaders can be supplied with the window-shader-fg option.
Supported values are either:

• "default" for the built-in default (just get the pixel)
• inline shader source in single quotes (')
• filename to a fragment shader, either absolute path or relative to the current directory (PWD when on cli, config file else) / in folder picom/shaders/ similar to the config-file search paths

The supplied shader must be a glsl fragment shader and at least consist of the function vec4 window_shader() which should return the current pixel for further processing (inversion, alpha-baking, brightness-clamping).

Window specific shaders can be specified with the window-shader-fg-rule option, similar to how opacity-rule works. Each entry is one of the supported values from window-shader-fg separated with a colon (:) from a window rule.
This currently comes with the limitation that the filename and inline source must not contain a colon.

Example configuration to desaturate unfocused windows:

#window-shader-fg = "default";
#window_shader-fg ="/etc/picom/shaders/testing/default.frag";
window-shader-fg-rule = [
    "desaturate.frag:focused != 1 && window_type = 'normal'",
]

And a file desaturate.frag next to the configuration file:

#version 330
in vec2 texcoord;
uniform sampler2D tex;

vec4 window_shader() {
    vec4 pixel = texelFetch(tex, ivec2(texcoord), 0);
    return vec4(vec3(dot(vec3(0.2126, 0.7152, 0.0722), pixel.rgb)), pixel.a);
}

Rendering Pipeline

Each specified shader is saved in a hash-map with a unique id. Each managed window has a reference to the configured shader in this hash-map and is updated with the reference of a window rule if specified. Else the configured default (window-shader-fg) or a fallback is used.
A hash-map is used to prevent loading the same shader multiple times and generating multiple programs for the same shader.

The OpenGL backend initializes programs for each shader in a new hash-map based on the shaders unique id. The gl_image struct keeps a reference to one entry in the program hash-map. The referenced program is used in _gl_compose() to render the image to screen.

The backend is extended with a new image operation IMAGE_OP_SET_SHADER. This is called on the image before compose() when the window uses a custom shader and should not be called on backends that do not support custom shaders (currently only the gl backend has support). The correct program is then selected from the hash-map (via id) and referenced in the image.

Discussion of new features

• Additional step in the rendering pipeline analogous to blurring with window-shader-bg. Render the background blur to a temporary texture and call a custom user-supplied shader on this and the original background. This may allow custom effects behind transparent windows or other deformations with background blur.

• Support for multiple shaders per rule by allowing multiple "shader objects" to be passed. This would allow modularized shaders for advanced use cases (common filters in separate files, custom composition for different rules, i.e. a separate color-key filter with different constants for different windows). This could probably be done by allowing multiple values separated by semi-colons (;) in the relevant config-options.

• The configuration options could be made more robust by explicitly stating default(), inline('#version 333…'), and file('shader.frag').

• Additional uniforms that might be useful:

  • Time/Counter for animations (currently implemented for glx-fshader-win)
  • Window position and window/texture size for custom scaling or translation
  • Access to the background texture?

• Should there be an option to use a completely custom shader and bypass the current pipeline completely?
  We currently depend on alpha-baking in the window shader. Color-inversion and brightness-clamping are features specified via user configuration. When we completely replace the window shader, the user must make sure to re-implement these features or else these options would loose their effect.

Todo

• Feature parity with existing glx-fshader-win
  • Add the time uniform for animations
  • Parse window-shader-fg option on the commandline
• Correctly protect against use on unsupported backends
• Load shader from file
• Implement new uniforms
• Add documentation to the manpage
• Support for multiple shaders / files
• Custom shaders for background rendering (after blur)

[absolutelynothelix]

i don't think that it's a good idea to leave the ability to pass a custom shader as string, the only sense is backwards-compatibility. the best way to pass a custom shader is to pass the path to a file containing a custom shader.

[tryone144]

Since the new option has a different name altogether, we shouldn't be concerned too much about backwards-compatibility.
Support for inline shaders was just easier for quick testing at the moment. We should be fine if we drop support for that and require file-based fragment shaders only. Other shader-interfaces are usually file-only as well.

[absolutelynothelix]

why we have to use window_shader function instead of the good-ol' and obvious main?

what do you think about implementing a file-watching mechanism similiar to the one implemented for the configuration file? when the file containing a custom shader is changed the shader is reloaded. it would be useful for developing and debugging shaders (especially complex ones). but i think it may have some performance hit when watching for, say, 10 shaders, so maybe it should be configurable for a particular shader.

[tryone144]

why we have to use window_shader function instead of the good-ol' and obvious main?

Because the custom shader currently doesn't replace the "default" rendering pipeline completely, but is just an extension. The default fragment shader still has the main() entry point. The win_shader_glsl (which contains main()) is linked into a single fragment shader with the one supplied via configuration or the default one.
I'm sorry for the wall-of-text in my initial post, but I tried to mention the rationale for that in the part for further discussion:

• Should there be an option to use a completely custom shader and bypass the current pipeline completely?
  We currently depend on alpha-baking in the window shader. Color-inversion and brightness-clamping are features specified via user configuration. When we completely replace the window shader, the user must make sure to re-implement these features or else these options would loose their effect.

File-watching should be theoretically possible. Currently, a change in one of the shaders would require completely destroying and re-initializing the backend like a config change already does. We could extend the backend-interface to separately initialize and destroy the shaders, depending on how tightly we want to couple this feature.
I'm not sure about the performance hit of multiple inotify file-watchers. If we make shader-watching optional, I would restrict this to a global level (i.e. watch all or none) and not shader specific.

[tryone144]

Another thing to consider: Are we fine with just a single pass for each window or do we want a pass for each shader?

The former wouldn't require complete redrawing of the window for each supplied shader while the latter would offer more freedom for the shaders (i.e. multiple "blur" passes) as each shader is self-contained and has access to the whole rendered image of the previous pass. If we opt to do multiple passes, how do we separate them in the config?

[codecov]

Codecov Report

Merging #507 into next will decrease coverage by 0.76%.
The diff coverage is 19.72%.

@@            Coverage Diff             @@
##             next     #507      +/-   ##
==========================================
- Coverage   37.72%   36.96%   -0.77%     
==========================================
  Files          46       46              
  Lines        9049     9307     +258     
==========================================
+ Hits         3414     3440      +26     
- Misses       5635     5867     +232     

Impacted Files	Coverage Δ
src/backend/gl/gl_common.c	0.00% <0.00%> (ø)
src/backend/gl/gl_common.h	0.00% <ø> (ø)
src/backend/xrender/xrender.c	0.00% <0.00%> (ø)
src/config.h	23.52% <ø> (ø)
src/string_utils.h	0.00% <ø> (ø)
src/win.h	41.66% <ø> (ø)
src/options.c	20.96% <17.24%> (-1.10%)	⬇️
src/config.c	33.57% <21.51%> (-8.47%)	⬇️
src/backend/backend.c	69.04% <33.33%> (-1.33%)	⬇️
src/config_libconfig.c	53.46% <41.66%> (-4.28%)	⬇️
... and 2 more

[tryone144]

Implemented shader file loading and removed support for inline shader source.
Added the time uniform to be "feature compatible" with the legacy backends.

We need a way to reliably mark windows with a custom shader as damaged (as the shader might change any of the window's pixels), mark them to be blended (to not rely on the force-win-blend option for all windows), and force redraw of specific windows at the refresh-rate for animations to work.

[tryone144]

See #295 (comment) for some additional considerations regarding the interface.

[yshui]

@tryone144

If we opt to do multiple passes, how do we separate them in the config?

Accept an array of file names?

[yshui]

@tryone144 hello! is it possible to revive this? I am thinking getting custom shader support into the new backends and enable them by default in v10.

[tryone144]

@yshui I am definitively open to putting more work into this. This was a bit on the back-burner, since I am not too fond of the way the window-shader-fg-rule option is implemented and got stuck on cleaning up the options / config interface (#662).

Some additional design considerations:

• [RFC] Custom window shader in experimental OpenGL backend #507 (comment):
  • "detect" alpha-blending, or force blending of windows when a custom shader is used?
  • forced-redraw for animations (using the time uniform)
  • (make sure damage tracking accounts for the whole window, especially in the above case)
• [RFC] Custom window shader in experimental OpenGL backend #507 (comment):
  • compile multiple shader-files into a single shader program
  • support multiple passes with different shaders programs
  • differentiate between the above in the configuration
    • not easily possible with the current rule implementation analogous to the opacity rules. Would benefit from a custom configuration section that could in turn easier adopted to work with the idea from Universal window matching #662.
• [RFC] Custom window shader in experimental OpenGL backend #507 (comment):
  • Keep parity with "default shader" that currently implements color-inversion, opacity, dimming, and rounded-corners.
    • Rely on main() in user-supplied shader and provide a default() function that does the above, which the user can call when they see fit?

[yshui]

@tryone144

• "detect" alpha-blending, or force blending of windows when a custom shader is used?

  I think we have force-win-blend for that.

• forced-redraw for animations (using the time uniform)

  Yeah we need to add this.

• Rely on main() in user-supplied shader and provide a default() function that does the above, which the user can call when they see fit?

  Or apply a user supplied function after the default one, or maybe use user shader in a separate pass from the default one? Maybe we can support all these. Possible design:

  custom-shader-passes = [ "default", "shaderfile1.glsl", "shaderfile2.glsl", ... ]

  Special keyword "default" means apply the default effects, which user can choose not to use.

  Do you think we need to do Universal window matching #662 first?

[kwand]

@yshui @tryone144

Perhaps looking into implementing #726 as part of this should be considered as well? (Or some sort of functionality that would offer easier support of applying a 3D LUT?)

I recently came across a tool on the Windows side that allows you apply a 3D LUT to the entire desktop, allowing proper color management for all apps for the first time (instead of the mess that is color management on Windows; though, a similar mess exists on Linux. Allegedly, Wayland has a protocol that allows for full-desktop color management. GNOME apparently plans to implement this protocol for Wayland and a similar feature for X with their Mutter compositor - though it's been stuck in the planning and PR stages for a while).

A hacky way to load the 3D lookup table would be to convert it to an inline constant table within the .glsl, though this is quite messy (a standard 65x65x65 3D LUT would require 274625 lines of code to recreate) and I'm not sure what performance impact this would have either.

It would probably be cleaner and easier to allow for the loading of a texture-format file (where it becomes the user's responsibility to convert from, for example, a 3D LUT filetype to a supported texture format file) or, if it's easier for now, implementing 3D LUT loading and application as a separate flag.

---

Adding 3D LUTs support for color management would be a great feature to add for the compositor. I'm told 3D LUT color management is superior to .icc profiles (which are allegedly only 1D LUTs) and to sRGB clamping on wide-gamut monitors (which results in banding, etc.; wide gamut is increasingly the standard for new monitors, resulting in very oversaturated colours for standard sRGB content).

I believe this would be the first time 3D LUTs would be supported on a standalone X compositor; the only other (Linux) compositor that supports this is apparently this plugin for compiz.

---

Edit: Looks like I mispoke. According to the README of dwm_lut, tetrahedral interpolation is needed when using a 3D LUT and they use a blue noise texture to reduce banding in SDR mode. Probably this will require a separate PR (if it is not possible to do within a .glsl), though certainly some of the code could be shared.

---

Second edit: There is this unmerged pull request into KWin/Plasma's compositor that may be useful as a reference (for ICC profile loading to create a 3D LUT - though I prefer the pipeline of using a dedicated LUT maker such as DisplayCal's 3DLUTMaker and supplying a .cube 3D LUT (same format used in Windows' dwm_lut)

[LoganDark]

Just reading the most recent comment here; does this mean picom will be able to perform proper color space conversions to display sRGB colors accurately on my Adobe RGB monitor? I'd be on board, as long as it's toggleable at runtime so I can take sRGB screenshots - currently my biggest problem with Windows color correction is that the correction shows up in screenshots.

Can't hide the entire compositor for screenshots though, because that would also disable transparency and blur and other effects!

[yshui]

@kwand Yes, custom texture would be a good thing to have, but as a first step I just want to get to feature parity with the old backend.

[yshui]

@tryone144 I am thinking maybe it would be better to have the custom shader take over everything, and provide the default processing as a utility function. This way, if the user still wants the default processing, it would just be one extra function call; if they don't they won't have to disable them one-by-one by using rules.

BTW, I am trying to rebasing this PR.

[LoganDark]

Sounds good to me

[yshui]

Hmm, I don't really like the fact a id -> shader mapping is maintained at 2 places.

[tryone144]

BTW, I am trying to rebasing this PR.

Okay. I would do that myself, but I'm still quite limited on time. If you need any hints, let me know.
Keep in mind this is indeed not polished at all and just about a working proof-of-concept.

Hmm, I don't really like the fact a id -> shader mapping is maintained at 2 places.

It's been a while, but where exactly are those stored?
IIRC, the configuration interface keeps a reference of all specified shaders (as in filepath and source code) and passes that to the backend to instantiate the actual shader programs. The backend itself needs to keep track of these to select the correct one for rendering a window and clean them up on backend-destruction.

[yshui]

@tryone144 There is one in config.c and keeps a map of path -> custom_shader, as well as a map tracking available ids. Then there is one in backend tracking id -> shader.

I am thinking adding a create_shader backend API, and let frontend track path -> shader.

[tryone144]

I am thinking adding a create_shader backend API, and let frontend track path -> shader.

As long as we keep this consistent across backend destruction and recreation, this might work. I think I used a specific integer id to reference the shaders instead of the path string to avoid sharing and duplicating strings between the configuration and the backend.

Letting the frontend track the appropriate backend shader-id returned by some create_shader API for the respective shader sounds good to me.

[yshui]

Superseded by #851