Fix Gradient Mask (Canvas) Inpaint Model Support

[dunkeroni]

Summary

Adds VAE and Image fields to the Create Gradient Mask node so that it can make a masked_latents tensor for inpaint models. Canvas inpaint/outpaint graphs are updated to include the new connections.

This behavior matches what the older Create Denoise Mask node did as well.

Related Issues / Discussions

Closes #5972

We've had some back and forth on using the UnetField to check for inpainting type and skip this encode process when it is not needed, but we need a reliable way to test for that on all model formats without loading the model itself (currently it is done by checking the in_channels number). If we come up with a standardized way to query this quality of the model, then we can add that and shave off a bit of time for canvas operations.

QA Instructions

Please test in Canvas interface, inpainting and outpainting. The inpaint models I have tried work, but we have seen in the past that some don't always behave like the others. Sometimes having a bit of edge_size on the coherence settings helps remove edge line artifacts, sometimes it creates different edge artifacts depending on the mask shape. For best results, start at zero and increase to 16 if necessary.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)

[hipsterusername]

Think it's the right decision to not overinvest in making inpainting optimized for speed, given these models are rarely used. The extra time seems worthwhile

[dunkeroni]

Think it's the right decision to not overinvest in making inpainting optimized for speed, given these models are rarely used. The extra time seems worthwhile

The optimization is not for inpainting models. It's for all other models. Because inpaint models require the extra VAE encode, we are performing that encode on all other models that don't require it. But right now we do not have a clean way of distinguishing between the two without an extra unet model load anyway.

So with this we lose a little bit of speed on all masked canvas operations, but we enable inpaint models to work again like they did before.

[psychedelicious]

In a discord conversation we discovered that during the MM2 refactor I had inadvertently removed the model variant enum from the diffusers model configs. The values were Normal (4 in-channels), Depth (5 in-channels, SD2.x only), Inpaint (9 in-channels).

To fix this:

• In invokeai/backend/model_manager/config.py, move variant: ModelVariantType = ModelVariantType.Normal from MainCheckpointConfig to MainConfigBase, from which both MainCheckpointConfig and MainDiffusersConfig inherit.
• In invokeai/frontend/web/src/features/modelManagerV2/subpanels/ModelPanel/ModelEdit.tsx, move the UI for variant select into a different conditional that only checks if the type is main, so that it is rendered for all main models.
• Update the model probe to correctly set the variant for diffusers models. I think this is already correct, but the variant field is getting stripped out. Fixing the classes as described above should automatically fix this.

Then I believe the changes int his PR could be much simpler.

Users with diffusers inpainting models installed before this fix is rolled out will have incorrectly-configured models. Here are a few ideas to fix existing models:

1. Advise users who encounter mushy inpainting to double check the setting in the model manager tab.
2. When loading a model for inference, if the number of in-channels is not correct for its variant setting, log a warning.
3. Write a migration that loads every main model, double-checks the in-channels and updates the db.

[dunkeroni]

Even with that, we would still need the changes in this PR (as they currently are) which allow the gradient mask node to produce a masked_latents.

If we add support for a quick Unet variant check, then we can add a unet field and only encode when necessary. Right now this PR matches how we previously supported inpaint models prior to 4.0 but now with gradient mask node.

[psychedelicious]

@dunkeroni Gotcha, thanks for clarifying that. I've updated this PR:

• Add the model variant to MainConfigBase, which makes both ckpt and diffusers models have this field
• Add a small test for the model probe to check that it detects properly (already did)
• Add a the variant field to the UI for diffusers models (cleaned up some UI components while I was there)
• Restore the UNet check in CreateGradientMaskInvocation
• Update the graphs to connect the UNet to the node
• Test all combinations of Canvas Inpainting/Outpainting, SD1.5/SDXL models, Inpainting/Normal model variants

Think we are good to go!