[Feature Request] Combing GroundingDINO and SAM for more automatic inpaint

[rentainhe]

Hello! Thanks for this great open-source project~

I think it will be more powerful to combine GroundingDINO and SAM and Lama-Cleaner together to build a powerful automatic inpaint system~

Here's our GroundingDINO and Grounded-SAM project:

• https://github.com/IDEA-Research/Grounded-Segment-Anything
• https://github.com/IDEA-Research/GroundingDINO

And I think it will be very interesting to build such functions~

[rentainhe]

I have another question about how to use the model defined in lama-cleaner locally like:

from lama_cleaner.model.lama import Lama

model = Lama()
model(image, mask)

Do you have some tutorial about this~ I runed it successfully but got nothing changed with the init image

[Sanster]

Hi, you can refer to this script as example. The meaning of hd_strategy in Config can be referred to here. https://github.com/Sanster/lama-cleaner/blob/37e9137ac70ef9021224a7c3037b21d35b835a01/lama_cleaner/schema.py#L7

import cv2
import PIL
import requests
import numpy as np
from lama_cleaner.model.lama import LaMa
from lama_cleaner.schema import Config


def download_image(url):
    image = PIL.Image.open(requests.get(url, stream=True).raw)
    image = PIL.ImageOps.exif_transpose(image)
    image = image.convert("RGB")
    return image


img_url = "https://raw.githubusercontent.com/Sanster/lama-cleaner/main/assets/dog.jpg"
mask_url = "https://user-images.githubusercontent.com/3998421/202105351-9fcc4bf8-129d-461a-8524-92e4caad431f.png"

image = np.asarray(download_image(img_url))
mask = np.asarray(download_image(mask_url).convert("L"))

model = LaMa("cpu")
result = model(image, mask, Config(hd_strategy="Original", ldm_steps=20, hd_strategy_crop_margin=128, hd_strategy_crop_trigger_size=800, hd_strategy_resize_limit=800))
cv2.imwrite("result.jpg", result)

[rentainhe]

Hi, you can refer to this script as example. The meaning of hd_strategy in Config can be referred to here.

https://github.com/Sanster/lama-cleaner/blob/37e9137ac70ef9021224a7c3037b21d35b835a01/lama_cleaner/schema.py#L7

import cv2
import PIL
import requests
import numpy as np
from lama_cleaner.model.lama import LaMa
from lama_cleaner.schema import Config


def download_image(url):
    image = PIL.Image.open(requests.get(url, stream=True).raw)
    image = PIL.ImageOps.exif_transpose(image)
    image = image.convert("RGB")
    return image


img_url = "https://raw.githubusercontent.com/Sanster/lama-cleaner/main/assets/dog.jpg"
mask_url = "https://user-images.githubusercontent.com/3998421/202105351-9fcc4bf8-129d-461a-8524-92e4caad431f.png"

image = np.asarray(download_image(img_url))
mask = np.asarray(download_image(mask_url).convert("L"))

model = LaMa("cpu")
result = model(image, mask, Config(hd_strategy="Original", ldm_steps=20, hd_strategy_crop_margin=128, hd_strategy_crop_trigger_size=800, hd_strategy_resize_limit=800))
cv2.imwrite("result.jpg", result)

Thanks a lot for your quick response! It works well! I will implement lama using lama-cleaner with grounded-sam in Grounded-Segment-Anything, it will be a very cool demo for the users.

[rentainhe]

BTW, I was wondering if lama-cleaner had already implemented the lama-refiner mentioned here: advimman/lama#112 or not~ @Sanster

[Sanster]

Refinement is not implemented, it needs to perform backward, which seems to have high requirements for GPU memory

[WaterKnight1998]

Refinement is not implemented, it needs to perform backward, which seems to have high requirements for GPU memory

@Sanster is implemented now??