Custom WGPU Shaders & Pipelines RFC

[bungoboingo]

Rendered

[hecrj]

Whoa, you really went down the rabbit hole!

I like the two first strategies the most (the widget could be built on top of the first one, I believe). The multiple backends seems to shift the complexity to the compositor, but adds a bunch of new ideas as well that seem hard to justify for now.

Overall, I feel we are focusing too much on the implementation details! The RFC focuses a lot on custom primitives and internals, but it's unclear how users will actually leverage this. What is the API that iced (the root level crate) will expose to allow implementing custom shaders? This is what the guide-level explanation is meant to tackle. Primitives, queues, backends, reflection, etc. are internal concepts the user should not have to worry about, ideally.

Think about a Canvas. You implement Program and use the Frame API to produce a list of Geometry and that's about it. Users are unaware of lyon, tessellation, mesh primitives, triangle pipeline, antialiasing, blit pipeline, etc. These are all internals that are part of the implementation, but are unnecessary ideas to use the widget.

That's what I would try to figure out first. How can a user render a rotating 3D cube inside of a widget? What is the simplest set of ideas the they will have to deal with? Do we have to expose the idea of a primitive? Or can we conveniently hide that? Do we have to expose wgpu? What will happen when the wgpu feature is disabled? Which approach is fun and clean?

Once we figure out what we want to offer, then we can focus on the implementation more. Basically, how do we make it happen?

[bungoboingo]

Updated this with more focus on the actual API exposed to users. I moved each design into its own file for ease of reading.

[hecrj]

Great! The design documents are the kind of stuff I expected!

The fn pointer approach forces users to implement both the Widget trait and a Renderable trait. You also seem to have to record the CustomPrimitive manually. The entry barrier is pretty high.

The custom backend approach exposes a bunch of concepts like backends and layers to the user. Let's try to avoid that as much as possible.

I think the shader widget approach makes the most sense. It's the simplest approach by far, with a single Program trait and it also feels familiar since Canvas uses a similar abstraction.

Now, let's iterate!

• Is Program::update called just when an Event occurs? What kind of Event? Any iced::Event? It's unclear what the purpose of this method exactly is since it both seems to deal with widget logic and drawing logic.
• Does Program::update need all those arguments? Transformation seems like an internal detail that shouldn't be leaked to the user.
• Why isn't Program using the prepare and render architecture? Providing a CommandEncoder to Program will force us to drop the current RenderPass for every Shader widget.
• Why does Shader::new take a fn pointer for init? How can an Application configure and change the behavior of a Shader? Imagine we wanted to go one step further and have a Slider that controls the size of the cubes. How do we allow for data to flow from the application state to the shader widget while keeping a single source of truth?
• Do we need to expose the idea of identifiers to the user? Is there a way we could avoid them? If we do need to expose the idea, then we most likely should have an opaque type that ensures no collisions.
• Can we just let the implementor of Program deal with tracking time? This way we can avoid introducing yet another argument to draw.
• Instead of RenderStatus, could we use the existing RedrawRequest enum?

These are some of the questions that arise when looking at the design. Bold part is maybe the most important! We will probably need to iterate further as we start polishing it!

[bungoboingo]

Is Program::update called just when an Event occurs? What kind of Event? Any iced::Event? It's unclear what the purpose of this method exactly is since it both seems to deal with widget logic and drawing logic.

I think that Program::update could be defined similarly (if not identically!) to Canvas::update, on some defined events like mouse events that are local to the custom_shader::Program. We could probably just re-use the same events that are defined for Canvas. We will need some way for the user to interact with events and mutate some internal state, which will also need to be defined as an associated type probably.

Does Program::update need all those arguments? Transformation seems like an internal detail that shouldn't be leaked to the user.

I agree that things could be removed. What exactly a user might need is a bit ill defined, but if we adjust update to only handle internal state mutations I think we can just keep it nearly identical to canvas::Program::update.

Why isn't Program using the prepare and render architecture? Providing a CommandEncoder to Program will force us to drop the current RenderPass for every Shader widget.

I think it could also include prepare as a method of custom::Program, though this will be another method we would need to dynamically dispatch. Do you think we should enforce layering in a custom pipeline like we do with quads, meshes, etc.?

Re: CommandEncoder vs RenderPass: I ran into a situation where I needed to use a new pass almost immediately when working with a 3D cube, which required a depth stencil. Sometimes a custom shader can use the current render pass, and sometimes it must drop the existing render pass & make a new one. What exactly the current status of the current render pass is it unclear to the user as it's redefined a few times during the rendering process. Maybe if we can guarantee a render pass has certain properties (targeting the window's surface texture, no depth stencil, loading & storing data, etc.) before we call render on their custom shader it might work. I think the correct solution is probably to let user's define their own render pass though since its parameters are going to vary custom shader to custom shader.

Why does Shader::new take a fn pointer for init? How can an Application configure and change the behavior of a Shader? Imagine we wanted to go one step further and have a Slider that controls the size of the cubes. How do we allow for data to flow from the application state to the shader widget while keeping a single source of truth?

I've updated Shader::new to just take a generic P: Program instead, which is all that's necessary. fn pointer was sort of lingering from an earlier iteration (see: design no 1). This makes it a lot more clear!

In terms of how an Application can configure and change the behavior of a custom Shader, I think adding methods for propagating user interaction down into the custom::Program itself similar to how we do it with Canvas will be all that's needed. Users can update their actual data being passed to the gpu in prepare which will get called every frame. If a Slider would need to control the size of the cubes, it could produce a message that would be used to update the shader, however the user wants to. An example (that I just wrote in this comment so might be typos lol):

// In an example application..
struct Example {
    cubes: Cubes,
    slider_value: f32,
}

enum Message {
    Cubes(CubesMessage),
}

impl Application for Example {
    fn update(&mut self, message: Message) -> Command<Message> {
        match message {
            Message::Cubes(cubes_msg) => {
                self.cubes.update(cubes_msg)
            }
        }
    }

    fn view(&self) -> Element<Message> {
        column![
            slider(1.0..=100.0, self.slider_value, Message::Cubes(cube::Message::CubeSizeChanged(self.slider_value))),
            self.cubes.view().map(Message::Cubes),
        ].into()
    }
}

mod cubes {
    pub struct Cubes {
        cube_size: f32,
        cubes_buffer: wgpu::Buffer,
        //..
    }
    
    enum Message {
        CubeSizeChanged(f32),
    }
    
   impl Cubes {
       pub fn view(&self) -> Element<CubeMessage> {
           custom::Shader::new(self)
               .width(Length::Fill)
               .height(Length::Fill)
               .into()
       }
       
       pub fn update(&mut self, message: Message) -> Command<Message> {
           match message {
               CubeSizeChanged(size) => {
                   self.cube_size = size;
               }
           }
       }
   }
   
   impl custom::Program<Message> for Cubes {
       fn prepare(
           &mut self,
           device: &wgpu::Device,
            queue: &wgpu::Queue,
            transformation: Transformation,
            scale: f32,
       ) {
           queue.write_buffer(&self.cubes_buffer, bytemuck::bytes_of(&Cubes::new(self.cubes_size)); //or whatever
       }
   }
}

I believe this should be fairly intuitive to users who have used a Canvas before.

Do we need to expose the idea of identifiers to the user? Is there a way we could avoid them? If we do need to expose the idea, then we most likely should have an opaque type that ensures no collisions.

No, we do not. After thinking about it more, since we are boxing the Program anyways, we could just make it Pin and use the pointer as the ID. Or we could just implement an internal hash when we create the Primitive::Custom. Or might not even need an ID at all, now that I'm thinking more about this! I'll do some more pondering.

Can we just let the implementor of Program deal with tracking time? This way we can avoid introducing yet another argument to draw.

Yes. This is a better solution than just always updating it in custom::Program::prepare (was in update before).

Instead of RenderStatus, could we use the existing RedrawRequest enum?

Yes! Can just make it Option<RedrawRequest> which will allow more flexibility anyways for users to only redraw at a certain time. They two enums were a bit redundant.

After all the feedback, a second iteration of this custom widget Program trait might look something like this:

pub trait Program<Message> {
    type State: Default + 'static;

    fn update(
        &self,
        _state: &mut Self::State,
        _event: custom::Event,
        _bounds: Rectangle,
        _cursor: custom::Cursor, //or true cursor availability..? soon(tm)?
    ) -> (event::Status, Option<Message>) {
        (event::Status::Ignored, None)
    }

    fn mouse_interaction(
        &self,
        _state: &Self::State,
        _bounds: Rectangle,
        _cursor: custom::Cursor,
    ) -> mouse::Interaction {
        mouse::Interaction::default()
    }

    fn prepare(
        &mut self,
        device: &wgpu::Device,
        queue: &wgpu::Queue,
        transformation: Transformation,
        scale: f32,
    );

    fn render(
        &self,
        _encoder: &mut wgpu::CommandEncoder,
        _device: &wgpu::Device,
        _target: &wgpu::TextureView,
        _scale_factor: f32,
        _target_size: Size<u32>,
        _bounds: Rectangle<u32>,
    ) -> Option<RedrawRequest> {
        None
    }
}

Where a user could make a custom::Shader with this definition:

pub struct Shader<Message, P: Program<Message>> {
    width: Length,
    height: Length,
    program: P,
}

impl<Message, P: Program<Message>> Shader<Message, P> {
    pub fn new(program: P) -> Self {
        Self {
            width: Length::Fill,
            height: Length::Fill,
            program,
        }
    }
    
    //...
}

I will need to think about how to pass that custom::Program into a Primitive::Custom enum variant in order to ensure proper ordering/layering/caching for reuse. I could move the bits that I need that aren't generic (e.g. prepare and render) into a separate trait that I can box and just use the pointer as the ID. I'll think about this more..

[bungoboingo]

Another thought I just had would be to force users to render into their own texture which is the size of their custom::Shader widget instead of the main surface's texture. This would encapsulate the whole design a bit more, and allow for some more flexibility. We would simply then need to take the texture they wrote to and sample it as we do any other image. Or could just use the image pipeline probably! This way we could cache it in the image atlas as well, and force users to call some kind of clear() similar to a canvas's Cache to re-render it. Might even be able to just re-use Primitive::Image instead of making a custom primitive variant, and add another data type to image::Handle like Data::Texture(&wgpu::TextureView). Would need to ensure it's rendered before it's composited with the surface. Could work? Haven't thought this through super well, but might be an interesting avenue to explore 🧐

[hecrj]

Awesome! Getting simpler!

I think it could also include prepare as a method of custom::Program, though this will be another method we would need to dynamically dispatch.

An additional dynamic dispatch call for a fairly rare primitive shouldn't be a big deal. Every single Element is a dynamic Widget. We potentially issue thousands of these calls per frame.

Do you think we should enforce layering in a custom pipeline like we do with quads, meshes, etc.?

I'd just render all the Shader widgets last in a layer for now.

I've updated Shader::new to just take a generic P: Program instead, which is all that's necessary. fn pointer was sort of lingering from an earlier iteration (see: design no 1). This makes it a lot more clear!

In the new example, how will Cubes be initialized (i.e. the wgpu::Buffer)?

In terms of how an Application can configure and change the behavior of a custom Shader, I think adding methods for propagating user interaction down into the custom::Program itself similar to how we do it with Canvas will be all that's needed.

Nested messages and update are not required to use Canvas. The new example currently duplicates the slider_value and the cube_size. Ideally, we should be able to call Cubes::view with the current size:

self.cubes.view(self.slider_value)

Message variants should not be generally accessed from outside of the module that defines them and, in this case, we only need the message to synchronize values.

Both these things seem to point towards the idea that a Program implementor should be the "virtual" state of the widget (i.e. the settings), while the State associated type should contain the persisted state (the wgpu stuff).

Sometimes a custom shader can use the current render pass, and sometimes it must drop the existing render pass & make a new one. What exactly the current status of the current render pass is it unclear to the user as it's redefined a few times during the rendering process.

All the pipeline primitives use the recommended approach by wgpu and take a RenderPass (except triangle), so it's already always guaranteed to be targetting the current frame.

Let's try to keep it simple for now. I know it reduces the scope and the capabilities considerably, but we can think about multiple rendering targets later.

[bungoboingo]

All the pipeline primitives use the recommended approach by wgpu and take a RenderPass (except triangle), so it's already always guaranteed to be targeting the current frame.

If we force the user to use our current render pass this is a potentially huge limitation. Any 3D shader for example would need to order its triangles back to front before rendering instead of using a depth buffer. The ability to create a shader with multiple passes (like for rendering water) would be impossible. Ideally I think we should have a system where users can piggy-back on to the existing render pass if they're doing more simple rendering work, or create their own. This could probably be a 2nd iteration, but something we should consider pretty soon afterwards. I can't think of any other method of doing this other than providing a second Program trait which has a &mut CommandEncoder in its render method vs &mut RenderPass in the original trait.

Maybe it would make sense to have a "simple" version of the custom shader program, and a more "advanced" custom shader program?

I'm working on an updated example for Cubes using the aforementioned agreed upon changes, will post back with answers to other questions!

[bungoboingo]

After a design convo today with @hecrj, the current implementation for the custom shader widget has been updated! These changes address the main issue that the original design had, which was that communicating across widgets required an intermediary message to sync states. Now things are much more seamless. You can see a new (rough) example in practice here.

Updated program trait:

/// The state and logic of a custom `Shader` widget.
///
/// A [`Program`] can mutate internal state and produce messages for an application.
pub trait Program<Message> {
    /// The internal state of the [`Program`].
    type State;
    /// The type of primitive this [`Program`] can render.
    type Primitive: custom::Primitive; //new!

    /// Update the internal [`State`] of the [`Program]. This can be used to reflect state changes
    /// based on mouse & other events. You can use the [`Shell`] to publish messages, request a
    /// redraw for the window, etc. which can be useful for animations.
    fn update(
        &mut self,
        _state: &mut Self::State,
        _event: Event,
        _bounds: Rectangle,
        _cursor: mouse::Cursor,
        _shell: &mut Shell<'_, Message>,
    ) -> event::Status {
        event::Status::Ignored
    }

    /// Returns the [`Primitive`] to be rendered.
    fn draw(&self, _state: &Self::State) -> Self::Primitive; // new!

    /// Returns the internal [`State`] of the [`Program`].
    fn state(&self) -> &Self::State; //TODO?

    /// Returns the current mouse interaction of the [`Program`].
    fn mouse_interaction(
        &self,
        _state: &Self::State,
        _bounds: Rectangle,
        _cursor: mouse::Cursor,
    ) -> mouse::Interaction {
        mouse::Interaction::default()
    }
}

The associated Primitive type now has a bounds of custom::Primitive, as defined here:

/// A set of methods which allows a [`Primitive`] to be rendered.
pub trait Primitive: Debug + 'static {
    /// Processes the [`Primitive`], allowing for GPU buffer allocation.
    fn prepare(
        &self,
        format: wgpu::TextureFormat,
        device: &wgpu::Device,
        queue: &wgpu::Queue,
        target_size: Size<u32>,
        storage: &mut custom::Storage, //new!
    );

    /// Renders the [`Primitive`].
    fn render(
        &self,
        storage: &custom::Storage,
        bounds: Rectangle<u32>,
        target: &wgpu::TextureView,
        target_size: Size<u32>,
        encoder: &mut wgpu::CommandEncoder,
    );
}

Similar to Renderable, this is where the user would actually define how to render their Primitive. The concept of storage here just refers to internal storage of pipelines, which (for now) is stored as a HashMap<TypeId, Box<dyn Any>> and reflected back to pipeline type T before the user interacts with it. This is a bit awkward at the moment and could use some work both ergonomics-wise & performance-wise in the final implementation.

See how a user might implement this custom::Primitive trait on their own Primitive type.

impl custom::Primitive for cubes::Primitive {
    fn prepare(
        &self,
        format: wgpu::TextureFormat,
        device: &wgpu::Device,
        queue: &wgpu::Queue,
        target_size: Size<u32>,
        storage: &mut custom::Storage,
    ) {
        if let Some(pipeline) = storage.get_mut_or_init::<Pipeline>(|| { // or something like this; it's awkward atm
            Pipeline::new(device, format, target_size)
        }) {
            pipeline.update(
                device,
                queue,
                target_size,
                &self.uniforms,
                self.cubes.len(),
                &self.raw_cubes(),
            );
        }
    }

    fn render(
        &self,
        storage: &Storage,
        bounds: Rectangle<u32>,
        target: &wgpu::TextureView,
        _target_size: Size<u32>,
        encoder: &mut wgpu::CommandEncoder,
    ) {
        if let Some(pipeline) = storage.get::<Pipeline>() {
            pipeline.render(target, encoder, bounds, self.cubes.len() as u32)
        }
    }
}

There is some leeway here for the user to violate the contract we give them, e.g. they could just choose to draw outside the bounds, but I think that is up to them whether or not they want to do that. By giving the user access to the wgpu::CommandEncoder we are allowing them pretty much free reigns to draw whatever they want anyways.

A user can implement this custom::Program on their own data type:

impl<Message> Program<Message> for Cubes {
    type State = ();
    type Primitive = cube::Primitive;

    fn draw(&self, _state: &Self::State) -> Self::Primitive {
        cube::Primitive::new(self.size, &self.origins, &self.camera, self.time)
    }
}

And create their custom shader like they would any other widget:

Shader::new(Cubes::new()) //or from some stored state!
      .width(Length::Fill)
      .height(Length::Fill),

You can see a working implementation of this design with this scuffed example:

cubes_cubes_cubes.mov