StableDiffusion variants

[gzsombor]

Hi,

I would like to ask about your opinion, if it is possible to add support to more stable diffusion variants?
I mean, as there are a bunch of checkpoints, base models out there for SD1.5, with a lot of loras as well, which could be used a bit more resource constrained environments, but currently, as far as I understand, carton only provides a single SDXL model. It would be nice, if one could combine everything together with carton 😉

Thanks!

[VivekPanyam]

This is definitely possible!

I think we need an easy way for people to create cartons with fine tuned and/or LoRA versions of stable diffusion. Maybe we provide a set of utilities that let people create cartons from stable diffusion variants in standard formats?

A basic version of this could be done entirely in Python code as a utility in the Python bindings.

We could package these models in a similar way to how we package transformers models (which is how SDXL is currently packaged).

Also, just based on the way Carton is designed, we'd be able to do pretty efficient storage of LoRA models. See here.

Is this something you'd be interested in working on?

Thanks!

---

Other thoughts:

One downside of this approach is that it uses python at runtime. We could try to use fully native stable diffusion implementations (like diffusers or pykeio-diffusers or candle). Integrating a native solution with LoRA support might give us the most flexibility (at the expense of taking longer to develop). Maybe something to explore in the future.

[VivekPanyam]

@gzsombor also, it's totally fine if you're not interested in working on this yourself. I would be curious for suggestions of stable diffusion variants to make available on the registry though