Using Regional Prompter with AnimateDiff's Prompt Travel (list object prompt support)

[SuperHimoYarouWorld]

Thank you with your great work!

For I plan to create short AI Animations with multiple people moving differently or looking different based on screenplay, I want Regional Prompter to associate with AnimateDiff's Prompt Travel. In other words, I need Stable Diffusion to play roles or present plays as if it were an entire troupe.

Testing Environment
OS: Linux 6.6.25-1-MANJARO
GPU: NVIDIA GeForce RTX 3060 VRAM 12G
CUDA Version: 12.3.107
Python Version: 3.11.8
PyTorch Version: 2.2.1+cu121
Gradio Version: 3.41.2
Webui-forge Version: f0.0.17v1.8.0rc-latest-276-g29be1da7
Regional Prompter Version: 50493ec
AnimateDiff(sd-forge-animatediff) Version: b20f7519
Dynamic Prompts Version: 1567e787

Test1 hack rp.py and regions.py
First, when I simply used Regional Prompter with AnimateDiff(with PromptTravel), I got the error messages below.

*** Error running process: ~/stable-diffusion-webui-forge/extensions/sd-webui-regional-prompter/scripts/rp.py
    Traceback (most recent call last):
      File "~/stable-diffusion-webui-forge/modules/scripts.py", line 803, in process
        script.process(p, *script_args)
      File "~/stable-diffusion-webui-forge/extensions/sd-webui-regional-prompter/scripts/rp.py", line 485, in process
        if OPTAND not in options: allchanger(p, "AND", KEYBRK)                                          #Change AND to BREAK
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "~/stable-diffusion-webui-forge/extensions/sd-webui-regional-prompter/scripts/rp.py", line 685, in allchanger
        p.prompt = p.prompt.replace(a, b)
                   ^^^^^^^^^^^^^^^^
    AttributeError: 'list' object has no attribute 'replace'

---

N.B. Without Prompt travel, Reagional Prompter worked well with AnimateDiff.
N.B. With sd-webui-dynamicprompts, there is No error like that.

According to these error messages, AnimateDiff with Prompt Travel gives list type object as positive or negative prompt. So I tried to add list type object support to rp.py and regions.py.

These belows are changes for rp.py and regions.py

• https://github.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/blob/main/SamplePatches/rp.py_adding_AnimateDiffPromptTravelSupport.patch
• https://github.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/blob/main/SamplePatches/regions.py_adding_AnimateDiffPromptTravelSupport.patch

(By the way, I'm not sure if it is correct or not since it is just a Machine-like replacement ;-p)

Then, I re-ran the webui, I got no such errors! So I think this hack may work with Attention Mode, at least.

Test2 Using AnimateDiff's Prompt Travel with Regional Prompter with "dynamic prompts extension"

Like below, I suddenly noticed that no error happened when I enabled sd-dynamic-prompts! HAHAHA! But why?

2024-04-14 18:40:19,839 - AnimateDiff - INFO - AnimateDiff process start.
2024-04-14 18:40:19,841 - AnimateDiff - INFO - Loading motion module mm_sd15_v3.safetensors from ~/stable-diffusion-webui-forge/extensions/sd-forge-animatediff/model/mm_sd15_v3.safetensors
2024-04-14 18:40:19,895 - AnimateDiff - INFO - Guessed mm_sd15_v3.safetensors architecture: MotionModuleType.AnimateDiffV3
2024-04-14 18:40:22,048 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 24 frames.
2024-04-14 18:40:22,049 - AnimateDiff - INFO - You are using prompt travel.
INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 24 images in a total of 1 batches.
1,1 0 Horizontal
Regional Prompter Active, Pos tokens : [29, 55], Neg tokens : [19]
2024-04-14 18:40:22,152 - AnimateDiff - INFO - Setting DDIM alpha.
2024-04-14 18:40:22,355 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet.

So, current most simple way with AnimateDiff's Prompt Travel is with using "dynamic prompts extension".

Animation Samples

These belows are sample Prompts and Results.

Regional Prompter Settings
Active = True
Generation Mode = Attention,Matrix
Base Ratio = 0
usebase = False
usecom = False
usencom = False
Main Splitting = Columns
Divide Ratio = 1,1
Width = 1024 #for the image size is 1024x576
Height = 576 #for the image size is 1024x576
flip","and";" = False
Overlay Ratio = 0.8 - 0.9
Presets: None
LoRA stop step = 0
LoRA Hires stop step = 0
LoRA in negative textencoder = 0
LoRA in negative U-net = 0
disable convert 'AND' to 'BREAK' = False
Use LoHa or other = False
Use BREAK to change chunks = False
debug = False
debug2 = True

with Regional Prompter

• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimations/00016-3418874118.webp
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimations/00152-752436473.webp
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimations/00218-1320549600.webp

Prompt

• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimations/00016-3418874118.txt
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimations/00152-752436473.txt
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimations/00218-1320549600.txt

without Regional Prompter (same seed, but prompt for each region was merged for AnimateDiff)

• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsWithoutRegionalPrompter/00003-3418874118.webp
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsWithoutRegionalPrompter/00004-752436473.webp
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsWithoutRegionalPrompter/00005-1320549600.webp

Prompt

• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsWithoutRegionalPrompter/00003-3418874118.txt
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsWithoutRegionalPrompter/00004-752436473.txt
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsWithoutRegionalPrompter/00005-1320549600.txt

I hope this idea helps you brush up your improvement of Regional Prompter.

Thank you.

[SuperHimoYarouWorld]

Test3 Latent Mode with Loras and Web UI

In Latent Mode with Loras and Web UI (not forge), Region Separation is correctly worked!

Results

• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsLatentMode/00016-495422161.webp
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsLatentMode/00038-1182408908.webp
  Prompt
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsLatentMode/00016-495422161.txt
• https://raw.githubusercontent.com/SuperHimoYarouWorld/UseAnimateDiffPromptTravelwithRegionalPrompterSample/main/SampleAnimationsLatentMode/00038-1182408908.txt
  Regional Prompter Settings are same as first post except using Latent Mode.

Although I got some problems below in Latent Mode, I satisfied the results. It'll give us the ability to perform new kind of visual expressions.

• Sometimes flipped Video's Left and Right.
• My direction was not clearly applied to each character's play, sometimes it fused.
• Loras effected a bit too strong.

Testing Environment
OS: Linux 6.6.25-1-MANJARO
GPU: NVIDIA GeForce RTX 3060 VRAM 12G
CUDA Version: 12.3.107
Python Version: 3.11.8
PyTorch Version: 2.1.2+cu121
Xformers version: 0.0.23.post1
Gradio Version: 3.41.2
Webui Version: 1.9.0
Webui arguments: --xformers --force-enable-xformers --xformers-flash-attention --medvram --opt-channelslast --no-half --no-half-vae --listen
Regional Prompter Version: 50493ec
AnimateDiff(sd-webui-animatediff) Version: 85a854b4
Dynamic Prompts Version: 1567e787

thanks a lot,