Resumable processing #430

pozitronik · 2023-06-11T08:11:23Z

#377
Much simpler approach than before (#408).
It is possible to determine the processing status from the state of the target temporary directory. The logic is simple: we mark already processed files by changing their names, so it is always known whether the processing has started and what remains.

This approach is also compatible with any types of processors. Also, you can change processing parameters on the fly without any consequences.

henryruhs · 2023-06-11T11:08:16Z

roop/core.py

-    update_status('Extracting frames...')
-    extract_frames(roop.globals.target_path)
+    # temp directory with frames is existed and all frames are extracted (or some already processed)
+    if state.is_exists(roop.globals.target_path) or state.unprocessed_frames_count(roop.globals.target_path) == detect_fc(roop.globals.target_path):


That needs to be one function like state.is_resumable()

henryruhs · 2023-06-11T11:09:20Z

roop/core.py

-    extract_frames(roop.globals.target_path)
+    # temp directory with frames is existed and all frames are extracted (or some already processed)
+    if state.is_exists(roop.globals.target_path) or state.unprocessed_frames_count(roop.globals.target_path) == detect_fc(roop.globals.target_path):
+        update_status(f'Temp resources for this target already exists with {state.processed_frames_cnt} frames processed, continue processing...')


Please do not use shortcuts in our code... there is no idx, cnt or err allowed...

henryruhs · 2023-06-11T11:10:13Z

roop/core.py

@@ -237,7 +244,7 @@ def start() -> None:


 def destroy() -> None:
-    if roop.globals.target_path:
+    if roop.globals.target_path and state.is_done(roop.globals.target_path) and state.unprocessed_frames_count(roop.globals.target_path) != detect_fc(roop.globals.target_path):


That needs to be a single function as awell like state.is_unfinished()

henryruhs · 2023-06-11T11:11:30Z

roop/enhancer.py

@@ -180,8 +183,8 @@ def multi_process_frame(source_img, frame_paths, progress) -> None:

 def process_video(source_path: str, frame_paths: list[str], mode: str) -> None:
    progress_bar_format = '{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}{postfix}]'
-    total = len(frame_paths)
-    with tqdm(total=total, desc='Processing', unit='frame', dynamic_ncols=True, bar_format=progress_bar_format) as progress:
+    total = len(frame_paths) + state.processed_frames_cnt


the state has to provide a method like state.count_unfinished_frames()

I tried to avoid the usage of global variables here. But it does not really matters.

henryruhs · 2023-06-11T11:13:02Z

roop/state.py

+
+
+def is_exists(target_path: str) -> bool:
+    """


Sorry but all the comments have to go... I reject to read them - if a method needs a paragraph to be explained, the method name is just bad.

The name is-exist does not tell me what exists... so I need to read your comment for sure. But this can actual be resolved by having a better name like has_unfinished_frames() or has_resumable_temp.

henryruhs · 2023-06-11T11:16:07Z

roop/swapper.py

@@ -52,7 +53,8 @@ def process_frames(source_path: str, temp_frame_paths: List[str], progress=None)
        temp_frame = cv2.imread(temp_frame_path)
        try:
            result = process_faces(source_face, temp_frame)
-            cv2.imwrite(temp_frame_path, result)
+            cv2.imwrite(state.get_frame_processed_name(temp_frame_path), result)


It would make send to add a variable that points to state.get_frame_processed_name(temp_frame_path) so the reader can easier understand what it contains

henryruhs · 2023-06-11T11:16:13Z

roop/enhancer.py

@@ -149,7 +151,8 @@ def process_frames(source_path: str, frame_paths: list[str], progress=None) -> N
        try:
            frame = cv2.imread(frame_path)
            result = process_faces(source_face, frame)
-            cv2.imwrite(frame_path, result)
+            cv2.imwrite(state.get_frame_processed_name(frame_path), result)


It would make send to add a variable that points to state.get_frame_processed_name(temp_frame_path) so the reader can easier understand what it contains

henryruhs · 2023-06-11T11:16:29Z

roop/swapper.py

@@ -90,8 +92,8 @@ def process_image(source_path: str, target_path: str, output_path: str) -> None:

 def process_video(source_path: str, temp_frame_paths: List[str], mode: str) -> None:
    progress_bar_format = '{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}{postfix}]'
-    total = len(temp_frame_paths)
-    with tqdm(total=total, desc='Processing', unit='frame', dynamic_ncols=True, bar_format=progress_bar_format) as progress:
+    total = len(temp_frame_paths) + state.processed_frames_cnt


Same like in enhancer

henryruhs · 2023-06-11T11:17:21Z

roop/utilities.py

@@ -34,6 +35,23 @@ def detect_fps(target_path: str) -> int:
    return 30


+def detect_fc(target_path: str) -> int:


Rename to detect_frame_count or get_total_frames

I think this belongs better to capturer.py and should not be done via ffmpeg but cv2.VideoCapture(video_path)

roop/state.py

henryruhs · 2023-06-11T11:19:45Z

roop/state.py

+from roop import utilities
+
+PROCESSED_PREFIX = '_'
+processed_frames_cnt = 0


Not a fan of this global counter... it should be calculated via a method everytime

Well, I would prefer to make this counter as a class attribute (probably accessed through a getter). But since classes are not used here, I used a global variable. Anyway, it doesn't matter.

henryruhs · 2023-06-11T11:20:47Z

roop/utilities.py

@@ -54,7 +72,8 @@ def restore_audio(target_path: str, output_path: str) -> None:

 def get_temp_frame_paths(target_path: str) -> List[str]:
    temp_directory_path = get_temp_directory_path(target_path)
-    return glob.glob(os.path.join(temp_directory_path, '*.png'))
+    return [os.path.join(temp_directory_path, file) for file in os.listdir(temp_directory_path) if


can we simplify this condition using any? I can barely understand this

pozitronik · 2023-06-11T12:56:11Z

roop/utilities.py

+    return [os.path.join(temp_directory_path, file) for file in os.listdir(temp_directory_path) if
+                not file.startswith(state.PROCESSED_PREFIX) and file.endswith(".png")]


Suggested change

return [os.path.join(temp_directory_path, file) for file in os.listdir(temp_directory_path) if

not file.startswith(state.PROCESSED_PREFIX) and file.endswith(".png")]

return [file for file in glob.glob(os.path.join(temp_directory_path, '*.png')) if not os.path.basename(file).startswith(state.PROCESSED_PREFIX)]

Is this better?

henryruhs · 2023-06-13T17:23:42Z

@pozitronik Please resolve the conflicts

pozitronik · 2023-06-13T18:14:11Z

@pozitronik Please resolve the conflicts

Done.
I've looked at the new code while merging and want to say it's excellent work! Keep it up!

# Conflicts: # roop/processors/frame/face_enhancer.py # roop/utilities.py

# Conflicts: # roop/utilities.py

* Rename landmark 5 variables * Mark as NEXT * Render tabs for multiple ui layout usage * Allow many face detectors at once, Add face detector tweaks * Remove face detector tweaks for now (kinda placebo) * Fix lint issues * Allow rendering the landmark-5 and landmark-5/68 via debugger * Fix naming * Convert face landmark based on confidence score * Convert face landmark based on confidence score * Add scrfd face detector model (#397) * Add scrfd face detector model * Switch to scrfd_2.5g.onnx model * Just some renaming * Downgrade OpenCV, Add SYSTEM_VERSION_COMPAT=0 for MacOS * Improve naming * prepare detect frame outside of semaphore * Feat/process manager (#399) * Minor naming * Introduce process manager to start and stop * Introduce process manager to start and stop * Introduce process manager to start and stop * Introduce process manager to start and stop * Introduce process manager to start and stop * Remove useless test for now * Avoid useless variables * Show stop once is_processing is True * Allow to stop ffmpeg processing too * Implement output image resolution (#403) * Implement output image resolution * Reorder code * Simplify output logic and therefore fix bug * Frame-enhancer-onnx (s0md3v#404) * changes * changes * changes * changes * add models * update workflow * Some cleanup * Some cleanup * Feat/frame enhancer polishing (#410) * Some cleanup * Polish the frame enhancer * Frame Enhancer: Add more models, optimize processing * Minor changes * Improve readability of create_tile_frames and merge_tile_frames * We don't have enough models yet * Feat/face landmarker score (#413) * Introduce face landmarker score * Fix testing * Fix testing * Use release for score related sliders * Reduce face landmark fallbacks * Scores and landmarks in Face dict, Change color-theme in face debugger * Scores and landmarks in Face dict, Change color-theme in face debugger * Fix some naming * Add 8K support (for whatever reasons) * Fix testing * Using get() for face.landmarks * Introduce statistics * More statistics * Limit the histogram equalization * Enable queue() for default layout * Improve copy_image() * Fix error when switching detector model * Always set UI values with globals if possible * Use different logic for output image and output video resolutions * Enforce re-download if file size is off * Remove unused method * Remove unused method * Remove unused warning filter * Improved output path normalization (s0md3v#419) * Handle some exceptions * Handle some exceptions * Cleanup * Prevent countless thread locks * Listen to user feedback * Fix webp edge case * Feat/cuda device detection (s0md3v#424) * Introduce cuda device detection * Introduce cuda device detection * it's gtx * Move logic to run_nvidia_smi() * Finalize execution device naming * Finalize execution device naming * Merge execution_helper.py to execution.py * Undo lowercase of values * Undo lowercase of values * Finalize naming * Add missing entry to ini * fix lip_syncer preview (#426) * fix lip_syncer preview * change * Refresh preview on trim changes * Cleanup frame enhancers and remove useless scale in merge_video() (s0md3v#428) * Keep lips over the whole video once lip syncer is enabled (s0md3v#430) * Keep lips over the whole video once lip syncer is enabled * changes * changes * Fix spacing * Use empty audio frame on silence * Use empty audio frame on silence * Fix ConfigParser encoding (#431) facefusion.ini is UTF8 encoded but config.py doesn't specify encoding which results in corrupted entries when non english characters are used. Affected entries: source_paths target_path output_path * Adjust spacing * Improve the GTX 16 series detection * Use general exception to catch ParseError * Use general exception to catch ParseError * Host frame enhancer models4 * Use latest onnxruntime * Minor changes in benchmark UI * Different approach to cancel ffmpeg process * Add support for amd amf encoders (#433) * Add amd_amf encoders * remove -rc cqp from amf encoder parameters * Improve terminal output, move success messages to debug mode * Improve terminal output, move success messages to debug mode * Minor update * Minor update * onnxruntime 1.17.1 matches cuda 12.2 * Feat/improved scaling (#435) * Prevent useless temp upscaling, Show resolution and fps in terminal output * Remove temp frame quality * Remove temp frame quality * Tiny cleanup * Default back to png for temp frames, Remove pix_fmt from frame extraction due mjpeg error * Fix inswapper fallback by onnxruntime * Fix inswapper fallback by major onnxruntime * Fix inswapper fallback by major onnxruntime * Add testing for vision restrict methods * Fix left / right face mask regions, add left-ear and right-ear * Flip right and left again * Undo ears - does not work with box mask * Prepare next release * Fix spacing * 100% quality when using jpg for temp frames * Use span_kendata_x4 as default as of speed * benchmark optimal tile and pad * Undo commented out code * Add real_esrgan_x4_fp16 model * Be strict when using many face detectors --------- Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com> Co-authored-by: aldemoth <159712934+aldemoth@users.noreply.github.com>

pozitronik mentioned this pull request Jun 11, 2023

Issue 377: support to pausing and resuming execution #408

Closed

pozitronik marked this pull request as draft June 11, 2023 08:25

pozitronik marked this pull request as ready for review June 11, 2023 09:24

henryruhs reviewed Jun 11, 2023

View reviewed changes

roop/state.py Show resolved Hide resolved

henryruhs reviewed Jun 11, 2023

View reviewed changes

pozitronik commented Jun 11, 2023

View reviewed changes

henryruhs changed the title ~~Issue 377 rethink~~ Resumable processing Jun 11, 2023

henryruhs force-pushed the next branch from c406665 to dcda604 Compare June 14, 2023 19:32

pozitronik added 6 commits June 15, 2023 18:41

issue #337: rebasing old commits to the new state of next branch

88f7f06

Merge branch 'next' into issue_377_rethink

2aeaf0b

Merge branch 'next' into issue_377_rethink

e08c235

# Conflicts: # roop/processors/frame/face_enhancer.py # roop/utilities.py

Merge branch 'next' into issue_377

d6cd54b

Satisfy CI check

9fa2680

Merge branch 'next' into issue_377

11a3b0b

# Conflicts: # roop/utilities.py

henryruhs deleted the branch s0md3v:next June 20, 2023 20:07

henryruhs closed this Jun 20, 2023

pozitronik deleted the issue_377_rethink branch June 29, 2023 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resumable processing #430

Resumable processing #430

pozitronik commented Jun 11, 2023 •

edited

Loading

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023 •

edited

Loading

henryruhs Jun 11, 2023 •

edited

Loading

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023 •

edited

Loading

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs Jun 11, 2023

pozitronik Jun 11, 2023

pozitronik Jun 11, 2023

henryruhs commented Jun 13, 2023

pozitronik commented Jun 13, 2023

		@@ -34,6 +35,23 @@ def detect_fps(target_path: str) -> int:
		return 30


		def detect_fc(target_path: str) -> int:

		return [os.path.join(temp_directory_path, file) for file in os.listdir(temp_directory_path) if
		not file.startswith(state.PROCESSED_PREFIX) and file.endswith(".png")]

	return [os.path.join(temp_directory_path, file) for file in os.listdir(temp_directory_path) if
	not file.startswith(state.PROCESSED_PREFIX) and file.endswith(".png")]
	return [file for file in glob.glob(os.path.join(temp_directory_path, '*.png')) if not os.path.basename(file).startswith(state.PROCESSED_PREFIX)]

Resumable processing #430

Resumable processing #430

Conversation

pozitronik commented Jun 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pozitronik Jun 11, 2023 • edited Loading

Choose a reason for hiding this comment

henryruhs Jun 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryruhs Jun 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryruhs commented Jun 13, 2023

pozitronik commented Jun 13, 2023

pozitronik commented Jun 11, 2023 •

edited

Loading

pozitronik Jun 11, 2023 •

edited

Loading

henryruhs Jun 11, 2023 •

edited

Loading

henryruhs Jun 11, 2023 •

edited

Loading