[1.1.447] PuLID

[huchenlei]

Overview

PuLID is an ip-adapter alike method to restore facial identity. It uses both insightface embedding and CLIP embedding similar to what ip-adapter faceid plus model does. However, there is an extra process of masking out the face from background environment using facexlib before passing image to CLIP. PuLID also uses Eva CLIP instead of normal CLIP. In the attn overrides, PuLID also does something more than IPAdapter, as it zero pads the tensor and adds to ortho of hidden states. If you are interested you can read their paper on how this is handled.

How to use

Install sd-webui-controlnet-evaclip extension

PuLID uses both evaclip embedding and insightface embedding as proj module input. So in order to use the feature, you need to install https://github.com/huchenlei/sd-webui-controlnet-evaclip extension.

Download Model

You can download fp16 model here: https://huggingface.co/huchenlei/ipadapter_pulid/resolve/main/ip-adapter_pulid_sdxl_fp16.safetensors

ControlNet Unit Setting

There is an extra radio group to let you select PuLID mode. According to my testing, they only make slight difference, but still something nice to have.

Example

Input

Output [Fidelity]:

Output [Style]:

Generation params

masterpiece painting, buildings in the backdrop, kaleidoscope, lilac orange blue cream fuchsia bright vivid gradient colors, the scene is cinematic, (1girl:1.2), portrait,, emotional realism, double exposure, watercolor ink pencil, graded wash, color layering, magic realism, figurative painting, intricate motifs, organic tracery, pol
Negative prompt: flaws in the eyes, flaws in the face, flaws, lowres, non-HDRi, low quality, worst quality,artifacts noise, text, watermark, glitch, deformed, mutated, ugly, disfigured, hands, low resolution, partially rendered objects, deformed or partially rendered eyes, deformed, deformed eyeballs, cross-eyed,blurry
Steps: 20, Sampler: Euler a, Schedule type: Automatic, CFG scale: 7, Seed: 3390099943, Size: 1024x1024, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Clip skip: 2, ControlNet 0: "Module: ip-adapter-auto, Model: ip-adapter_pulid_sdxl_fp16 [d86d05ea], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Hr Option: Both, Save Detected Map: True", Version: v1.9.3-2-g3d3fc81f

Multiple inputs

Note

Comparing to instant id, you need stronger prompt to make the output stylized. Following is the same neon style generated by instant id:

Known issue

facexlib and evaclip in total consumes about 2~3GB of vram. So if you have low vram, that can be an issue. There is also about 500MB of vram not unloaded after unloading the preprocessors. I will need to investigate more on why vram here is not properly freed.

[songispm]

Error in Preprocessor, Please help.

[songispm]

Ah, change the code to this, and it worked. @huchenlei

from PIL import Image

face_features_image = resize(
    input_image,
    resize_target,
    Image.BILINEAR,
)

[huchenlei]

Interesting. I think torch has changed something in higher version? My torch version is 2.1.2+cu122

[songispm]

It's very very very good, thank you

[songispm]

Ok, I can't wirte python, I just search it.

Then, Please help to make it the most compatible way.

[huchenlei]

Done. huchenlei/sd-webui-controlnet-evaclip@d850909

[songispm]

another question:

error in image about complete 2D, flat, line art,
Please help optimize it. (not a hurry, but need it)

sometimes
Exception： Insightface: No face found in image.
sometimes
RuntimeError: facexlib align face fail

[huchenlei]

The input face must be realistic. These face detection lib cannot handle stylized face.

[digiconn]

=====
ValueError: 'PuLIDMode.FIDELITY' is not a valid PuLIDMode

Seeing` this error both for SD 1.5 and SDXL models while using A1111. And this error appears both for "ip-adapter_face_id_plus" and "ip-adapter_pulid".

[huchenlei]

See #2844

[digiconn]

Thank you, its fixed. And results are beautiful. I am wondering if you can have either "Fidelity" or "Extremely Style" selected on load. Unless we choose one, the model doesn't apply and its confusing user experience. Thanks again :)

[Soulreaver90]

Anytime I copy an old generation into txt2img that used faceidplusv2, I get ValueError: 'PuLIDMode.FIDELITY' is not a valid PuLIDMode
I am on the latest version of controlnet and A1111.

[nvm01]

Any idea if this can be implemented in ControlNet?
https://github.com/HVision-NKU/StoryDiffusion
This is really close to what this last ID makes.
Thanks!

[JetsprintRacer]

So... I assume, that this thing is only for powerful GPU's...? It took almost 13 minutes to generate image, based on a 3 image Multi-input on RTX 3080 Ti. Also, multi-input is only enough for one generation. Second attempt ends with OutOfMemoryError: CUDA out of memory.
Some single image generations are also slow.

[huchenlei]

These preprocessors (facexlib and evaclip) seem to have vram leak. See ToTheBeginning/PuLID#16

[pooria98]

Yeah I also have the same problem (RTX 3060). with a lightning model, I generated an image in 7 seconds but it took 7 minutes with pulid (with same seed and settings). Other generations after that with disabled controlnet also take the several minutes. it messes up the gpu until I restart my pc.
how do people use this? is everyone ok with this???

[a3nima]

thanks for your work @huchenlei . pulid works well resambling the face and impression from the initial photo(s), but it seems - contrary to face id - you cannot change that (for example: smiling, laughing etc). also the eye color did not change with prompting in my first test.

[huchenlei]

Please file this issue in PuLID repo: https://github.com/ToTheBeginning/PuLID

Personally I think PuLID is a very strong model which can be less flexible.

[yanghongquan521]

Which folder to put it in EVA02_CLIP_L_336_psz14_s6B.pt

[a3nima]

in A1111 this gets downloaded and installed by itself when running pulid for the first time.

[roelfrenkema]

git clone it in the extension dir

[Goldsmith1a]

Error Please Help me

[IAFFeng]

Same problem, please help

[JIAOJIAYUASD]

using PuLID and get a bug like above: ModuleNotFoundError: No module named 'onnxruntime'

[JIAOJIAYUASD]

this bug occur when I update the insightface. I pip install onnxruntime-gpu to fix it.

[aminesoulaymani]

Multi-input doesn't seem to work, the module use in fact the image from the single-input tab (if there is one already set).
To reproduce: restart the module, refresh navigator, put some images in the multi-input tab (nothing in the single input tab), generate

[KingSnailtan]

Yes, I can attest that it doesnt seem to work. Using SD15. Preprocessor on auto and plusv2 and portrait tested.
Always same error that no faces are found

[JIAOJIAYUASD]

webui-forge does not support PuliD, use a1111 webui发自我的荣耀手机-------- 原始邮件 --------发件人： Hoodad Mehrbod ***@***.***>日期： 2024年5月9日周四 傍晚6:52收件人： Mikubill/sd-webui-controlnet ***@***.***>抄送： JIAOJIAYUASD ***@***.***>, Comment ***@***.***>主 题： Re: [Mikubill/sd-webui-controlnet] [1.1.447] PuLID (Discussion #2841) it doesnt work on forge...any one knows why? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[Beckhamdaz]

Hi everyone! I need help

please help me!!

[huchenlei]

Currently PuLID only works for SDXL. The author has not released any SD15 models yet.

[CubieX]

How is this different from the IP-Adapter FaceId method?
Or to be more precise: Which cases does it handle better than FaceID?

[huchenlei]

It has better fidelity, but you get less control on facial expression

[OoAnikioO]

After several tests, the eyes are often completely failed, and the skin is very smooth, not too realistic.

[CaptailTyler]

It runs very slow on my PC, as does any SDXL models. Would be great to have the option of SD 1.5 too.

[TigerHH6866]

any plan for 1.5??

[wangzhngfs]

What's the problem with this? How to solve it?

[wangzhngfs]

懂得朋友帮忙解决以下，谢谢

[zhouyi531]

Great work! Do you mind also introduce how to use it via API?

[changpeidong]

预览错误，生成屏幕什么问题？

[zzz12345zzz]

我也是这个问题

[greengerong]

get error:

Error running process: /mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/mnt/d/project/stable-diffusion-webui/modules/scripts.py", line 825, in process
        script.process(p, *script_args)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1222, in process
        self.controlnet_hack(p)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1207, in controlnet_hack
        self.controlnet_main_entry(p)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 941, in controlnet_main_entry
        controls, hr_controls, additional_maps = get_control(
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 290, in get_control
        controls, hr_controls = list(zip(*[preprocess_input_image(img) for img in optional_tqdm(input_images)]))
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 290, in <listcomp>
        controls, hr_controls = list(zip(*[preprocess_input_image(img) for img in optional_tqdm(input_images)]))
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 242, in preprocess_input_image
        result = preprocessor.cached_call(
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/supported_preprocessor.py", line 196, in cached_call
        result = self._cached_call(input_image, *args, **kwargs)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/utils.py", line 82, in decorated_func
        return cached_func(*args, **kwargs)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/utils.py", line 66, in cached_func
        return func(*args, **kwargs)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/supported_preprocessor.py", line 209, in _cached_call
        return self(*args, **kwargs)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/preprocessor/ip_adapter_auto.py", line 25, in __call__
        return p(*args, **kwargs)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/preprocessor/pulid.py", line 152, in __call__
        face_features_image = self.facexlib_detect(input_image)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/preprocessor/pulid.py", line 122, in facexlib_detect
        return facexlib_preprocessor(input_image, return_tensor=True)
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/preprocessor/pulid.py", line 70, in __call__
        self.load_model()
      File "/mnt/d/project/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/preprocessor/pulid.py", line 32, in load_model
        self.model = FaceRestoreHelper(
      File "/mnt/d/project/stable-diffusion-webui/venv/lib/python3.10/site-packages/facexlib/utils/face_restoration_helper.py", line 99, in __init__
        self.face_det = init_detection_model(det_model, half=False, device=self.device, model_rootpath=model_rootpath)
      File "/mnt/d/project/stable-diffusion-webui/venv/lib/python3.10/site-packages/facexlib/detection/__init__.py", line 22, in init_detection_model
        load_net = torch.load(model_path, map_location=lambda storage, loc: storage)
      File "/mnt/d/project/stable-diffusion-webui/modules/safe.py", line 108, in load
        return load_with_extra(filename, *args, extra_handler=global_extra_handler, **kwargs)
      File "/mnt/d/project/stable-diffusion-webui/modules/safe.py", line 156, in load_with_extra
        return unsafe_torch_load(filename, *args, **kwargs)
      File "/mnt/d/project/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 815, in load
        return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
      File "/mnt/d/project/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 1051, in _legacy_load
        typed_storage._untyped_storage._set_from_file(
    RuntimeError: unexpected EOF, expected 4946769 more bytes. The file might be corrupted.

config:

How to fix it?

[luochenghua0]

在SD中根本不能用

[IAFFeng]

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: F:\sd webui\webui\extensions\sd-webui-controlnet\annotator\downloads\insightface\models\antelopev2\glintr100.onnx recognition ['None', 3, 112, 112] 127.5 127.5
2024-06-17 16:43:35.9224418 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "F:\sd webui\system\python\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

[roelfrenkema]

This is a genius extension. Thank you very much I love it.

[vincefield1130]

I'm getting the following error message, kindly help: @huchenlei

Model loaded in 9.1s (load config: 1.8s, create model: 1.5s, apply weights to model: 5.4s, move model to device: 0.1s, calculate empty prompt: 0.1s).
2024-07-01 17:15:44,587 - ControlNet - INFO - unit_separate = False, style_align = False
*** Error running process: /content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/scripts.py", line 825, in process
script.process(p, *script_args)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1222, in process
self.controlnet_hack(p)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1207, in controlnet_hack
self.controlnet_main_entry(p)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 890, in controlnet_main_entry
Preprocessor.unload_unused(active_processors={
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/supported_preprocessor.py", line 181, in unload_unused
f"Unload unused preprocessors. Active: {[p.name for p in active_processors]}"
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/supported_preprocessor.py", line 181, in
f"Unload unused preprocessors. Active: {[p.name for p in active_processors]}"
AttributeError: 'NoneType' object has no attribute 'name'

[xlc-github]

same error

[vincefield1130]

Forgot to update, I actually forgot to install the evaclip extension. After install it, the control net works okay, but the outcome image wasn't as impressive as the one in the demo above. Trying to figure out the settings.

[xlc-github]

how to solve this error!
I have already git the sd-webui-controlnet-evaclip to extensions

[venomsoft1]

1. Git clone fresh sd-webui-controlnet-evaclip to extensions if you changed the code

2. Go to /extensions

3. Rename "sd-webui-controlnet-main" folder to "controlnet"

4. Go to sd-webui-controlnet-evaclip/scripts and open the file "preprocessor_evaclip.py" with notepad, IDE or any code editor. I use Visual Studio Code because it has lots of opportunities for coders as me

❗5. Change line 14 as shown in screen_1 and save the file. I wrote new path from controlnet to evaclip

6. Run A1111. Set the PuLID in Controlnet

7. Generate or click the "💥" button. After that A1111 will download the EVA02-CLIP preprocessor (size:~800MB)

Done.

------------*
OPTIONAL: You can make sure that all the needed libraries are installed

1. Open cmd

2. Write: cd \Scripts

3. Write: activate

4. Then check that libraries are installing by writing this one by one:

pip install insightface
pip install pydantic
pip install albumentations
pip install onnxruntime
pip install pip install onnxruntime-gpu

5. Write: deactivate

6. Run A1111 and use PuLID

Done.

[lior007]

WHAT IS THE FOLDER LOCATION TO PUT THE FILE IN THE COMFUI MODEL FOLDER?: ip-adapter_pulid_sdxl_fp16.safetensors ?

[luochenghua0]

·comfyui/models/pulid/

…

---Original--- From: ***@***.***> Date: Fri, Jul 26, 2024 06:03 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [Mikubill/sd-webui-controlnet] [1.1.447] PuLID (Discussion #2841) WHAT IS THE FOLDER LOCATION TO PUT THE FILE IN THE COMFUI MODEL FOLDER?: ip-adapter_pulid_sdxl_fp16.safetensors ? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[ClothingAI]

1. Does it work with SD_NEXT?
2. Same for Forge?

[zethfoxster]

looks like they set it up for normal control net, not the integrated versions forge/sdnext....wish they consider switching to forge support since auto1111 hasn't updated in 6 months.....