the image not matching text prompt for depth control inpainting

[renrenzsbbb]

Thanks for your great work. I test your code for your demo img, the result is good. however, I test it for depth control inpainting, the output image is not match with text prompt. Can you give me some advices.
first, I test the depth controlnet in original controlnet, the prompt is "a firefighter wearing red uniform clothes". the result is good.

the above image is input for depth control net.

the above image is depth prediction map by control net.

the above image is output by control net.
second, I add a mask image to control net, and use the same input image and text prompt, depth map from control net. However, the output image is not matched with text prompt. the clothes color is not red.

the above is mask img.

the above is final output by your proposed code.

[haofanwang]

Well, this is because of the UNet. To support inpainting, we have to load the UNet from inpainting model, instead of the original UNet in ControlNet. Also, we cannot use other better base model here.

So this is a known limitation, we are trying to train a inpainting model with ControlNet. @renrenzsbbb

[renrenzsbbb]

Thanks for your quick reply. Can you provide timeline about adding inpaint model to control net.