Immich Processing Library Item multiple times

[psla]

Pre-submission Checks

• I checked for similar issues, but could not find any. I also checked the closed issues. I could not contribute additional information to any existing issue.
• I will take the time to fill in all the required fields. I know that the bug report may be dismissed otherwise due to lack of information.

Describe the bug

I refreshed my library in immich after fixing issue with glob patterns. Unfortunately, the thumbnails do not seem to get processed, or get processed extremely slowly (after a week I am still 300k short of all files, and I have been 300k short 5 days ago too).

While some progress is being made, Immich attempted to parse a folder containing 11k files approx 130k times. Here is a quick analysis of a single file, /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG

There are 32 entries for this file in the docker log:

grep 2021-09_Italy_Wlochy /mnt/newhome/unprivileged-docker/.local/share/docker/containers/c374ef9a2f3b9cf3548b2877d10223659a85497898e67db7b6ec904df367e85a/c374ef9a2f3b9cf3548b2877d10223659a85497898e67db7b6ec904df367e85a-json.log  > ~/logs.gz

unprivileged-docker@usa:~$ grep 2021-09_Italy_Wlochy/D750/DSC_0165.JPG ~/logs  | wc -l
32

It looks like this file was queued for metadata extraction 5 times

{"log":"\u001b[95m[Nest] 7  - \u001b[39m09/15/2024, 11:15:31 PM \u001b[95m  DEBUG\u001b[39m \u001b[33m[Microservices:LibraryService]\u001b[39m \u001b[95mQueueing metadata extraction for: /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-15T23:15:31.652787541Z"}
{"log":"\u001b[95m[Nest] 7  - \u001b[39m09/16/2024, 12:31:23 AM \u001b[95m  DEBUG\u001b[39m \u001b[33m[Microservices:LibraryService]\u001b[39m \u001b[95mQueueing metadata extraction for: /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-16T00:31:23.466406573Z"}
{"log":"\u001b[95m[Nest] 7  - \u001b[39m09/18/2024, 12:33:03 AM \u001b[95m  DEBUG\u001b[39m \u001b[33m[Microservices:LibraryService]\u001b[39m \u001b[95mQueueing metadata extraction for: /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-18T00:33:03.140897797Z"}
{"log":"\u001b[95m[Nest] 7  - \u001b[39m09/20/2024, 12:24:32 AM \u001b[95m  DEBUG\u001b[39m \u001b[33m[Microservices:LibraryService]\u001b[39m \u001b[95mQueueing metadata extraction for: /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-20T00:24:32.966181535Z"}
{"log":"\u001b[95m[Nest] 7  - \u001b[39m09/22/2024, 12:25:23 AM \u001b[95m  DEBUG\u001b[39m \u001b[33m[Microservices:LibraryService]\u001b[39m \u001b[95mQueueing metadata extraction for: /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-22T00:25:23.091309452Z"}

And the preview was also generated 4 times

{"log":"\u001b[32m[Nest] 7  - \u001b[39m09/17/2024, 8:21:07 AM \u001b[32m    LOG\u001b[39m \u001b[33m[Microservices:MediaService]\u001b[39m \u001b[32mSuccessfully generated JPEG image preview for asset /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-17T08:21:07.052349783Z"}
{"log":"\u001b[32m[Nest] 7  - \u001b[39m09/19/2024, 2:25:58 PM \u001b[32m    LOG\u001b[39m \u001b[33m[Microservices:MediaService]\u001b[39m \u001b[32mSuccessfully generated JPEG image preview for asset /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-19T14:25:58.203703109Z"}
{"log":"\u001b[32m[Nest] 7  - \u001b[39m09/20/2024, 8:03:28 PM \u001b[32m    LOG\u001b[39m \u001b[33m[Microservices:MediaService]\u001b[39m \u001b[32mSuccessfully generated JPEG image preview for asset /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-20T20:03:28.94949771Z"}
{"log":"\u001b[32m[Nest] 7  - \u001b[39m09/22/2024, 5:22:55 PM \u001b[32m    LOG\u001b[39m \u001b[33m[Microservices:MediaService]\u001b[39m \u001b[32mSuccessfully generated JPEG image preview for asset /mnt/media/Fotografie/2021/2021-09_Italy_Wlochy/D750/DSC_0165.JPG\u001b[39m\n","stream":"stdout","time":"2024-09-22T17:22:55.154199538Z"}

The log file from docker has now over 4GB with 13M lines.

It seems like immich is processing some of the same files multiple times and not getting through the backlog of the files to process.

logs.gz

Have you seen this before? Is there anything else I can do to help diagnose the issue?

It does seem like Immich is making some progress, as the number of "outstanding" files reduced in the past 3 days, but it feels like it's making a progress slowly, processing each file multiple times.

Expected behavior

Process each file once

Steps to reproduce the issue

not sure.

1. I changed the "excluded path" patterns in my external library (one that was processed before)
2. And then I refreshed (scan library) operation

I've waited a week and it seems like immich got stuck, not making much progress, and recent files do not have thumbnails generated for them (some do, some don't)

Screenshots

Screenshot today:

Screenshot 3 days ago

Edit! Screenshot now (a few hours later). The backlog is back to 320k items. and FWIW no Library tasks or metadata tasks are being performed (i.e. I am not adding tens of thousands of new files)

Logs

logs.gz

Happy to provide more, the file is massive though (4GB, might compress to a few hundred megs)

Client version number

n/a (web client)

Desktop environment (Linux only)

Linux 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux

Client package version and origin (Linux only)

No response

Installation path (Windows only)

No response

Server information

1.115.0, docker

Additional context

No response

[psla]

Wondering if periodic library scanner would re-queue files that are already in the queue.... Leading to processing them multiple times.

I just disabled it to see if it will help.

[DeepDiver1975]

Looks like you opened this issue in the wrong repository.