Scan & offline assets issue - not offline and not scanning

[matticas]

Hi, this issue has been since ~1.100? I think. I started Immich around 1.9x ? I am now on 1.105.1

I think the issue is with my old Portainer-Stack file settings??

Symptoms are;

1. All my External Library Assets are "offline".
2. Scan, Forced Re-scan all do nothing on external library. I get 2 seconds of CPU usage, then nothing, and no changes.

I have checked read-only access within both containers and they work with the external libraries;

• immich-server: upload/
• immich-server: /My_Photos/
• immich-microservices: upload/
• immich-microservices: /My_Photos/

All four of those photos have read-only access.

I have checked the external paths, and validated them successfully.

The rest of Immich works great, including new images from 2x Android phones nightly, including facial recognition etc.

But, I am unsure how to update the PG database from docker volume to a mounted volume? ( I don't want to loose all my albums and face tagging for 20yrs of images).

My Settings;

Portainer Stack;

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/Shed4/My Photos/:/My_Photos:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/Shed4/My Photos/:/My_Photos:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

The latest Portainer Stack; (image paths, and namely the volume for database container)

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]

Would this be causing my scan and offline issues?

How do I migrate the docker volume to my mounted disk volume?

Thanks!

[alextran1502]

All my External Library Assets are "offline".

Can you check if the mount is actually there?

[psla]

Related: #6845

In my case it is. I have the same problem. All the files (that I tested manually) are there, and accessible from the container itself.

In my case regenerating thumbnail and regenerating metadata works fine too, but the asset stays "offline"

[Nest] 17  - 06/28/2024, 3:31:48 AM   DEBUG [Api:LoggingInterceptor~1muc0uxc] POST /api/assets/jobs 204 2.28ms 10.0.7.76
[Nest] 7  - 06/28/2024, 3:31:48 AM     LOG [Microservices:MediaService] Successfully generated JPEG image preview for asset d48f4516-a020-4f1c-968a-043df9c48715
[Nest] 7  - 06/28/2024, 3:31:48 AM   DEBUG [Microservices:MediaService] Deleting old preview for asset d48f4516-a020-4f1c-968a-043df9c48715
[Nest] 7  - 06/28/2024, 3:31:49 AM     LOG [Microservices:MediaService] Successfully generated WEBP image thumbnail for asset d48f4516-a020-4f1c-968a-043df9c48715

[psla]

In my case the issue was an exclusion pattern. More details in #6845

[matticas]

Hi Alex,

Firstly, thanks for Immich!

This is how I check; (correct me if i'm wrong?)

Host Terminal;

docker exec -it immich_server bash
cd upload/
ls -l   <-- can see contents of external library
cd /My_Photos
ls -l   <-- can also see contents of external library
exit
docker exec -it immich_microservices bash
cd upload/
ls -l   <-- can see contents of external library
cd /My_Photos
ls -l   <-- can also see contents of external library
exit

Is this the correct way to check it?

Edit;

I also check the microservices container logs; and notice this after every scan, re-scan, and forced re-scan;

DEBUG [ImmichMicroservices] [LibraryService] Found 0 asset(s) when crawling import paths /My_photos/import/path/one,/My_photos/import/path/two,/My_photos/import/path/twenty

This always comes up as 0 assets, even if they are present?

Is it a permission thing with the docker containers? I don't know what else to check - it worked great for all the initial imports of external library, and i'm not sure when it all went 'offline'.

[matticas]

@alextran1502 any ideas? Anything I can try to investigate?

I'm really trying to avoid making a new library; 2 weeks of scanning, 24 import paths, 175 albums, 210 named people... i really wish to not do all that again.

[matticas]

Just ran another test;

Created a new Import Library using the gui, the folder was nested in my external library called "test_import";

Both server and microservices containers can see it in the 4 locations;

• immich-server: upload/test_import/
  

• immich-server: /My_Photos/test_import/
  

• immich-microservices: upload/test_import/
  

• immich-microservices: /My_Photos/test_import/
  

However, the microservices container "Found 0 assets when crawling import paths" but didn't list the new import path; "test_import".

Is the external library scanning feature having bugs?

[matticas]

Hi etony;

The containers can definitely see the import paths. I've checked this again just now, and as above.

The test_import directory above now works, so scanning is working. But the original large external library isn't been detected.

What permissions should the container see? I just did a sweep through, and i'm getting mixtures of 777, 664, and 700. Where do I set these? (my compose file has the mounts as Read-Only for now).

Thanks!

[etnoy]

Do you see a green check box next to the import path in the web ui?

[matticas]

Yes, on both the original external library, and the new test one, using the Validate button.

[matticas]

Another test just now;

My original external library has 24 import paths.

I just made a new external library, and added the first import part which only has 2 photos. It validated and scanned successfully, and the assets are not offline.

Thus, I suspect the issue is with the database and the original external library? microservices can definitely see the directories. What else can I do to inspect the database? (i've never used a database).

ta

[etnoy]

The core issue must be that the import path is not printed in the logs.

If you enable debug logging, does it print more info? What happens if you remove, save, and readd the path? Assets won't be deleted so don't worry

[matticas]

@etnoy
(sorry replied here, not in thread)

So;

Debug & Verbose logging have been on for some time (i'm watching the microservices container logs). New External Libraries get listed, all the actions get listed, how many assets it finds, faces, etc.

The original External Library that plays up keeps displaying this;

[Nest] 7  - 05/21/2024, 6:46:55 AM     LOG [ImmichMicroservices] [LibraryService] Refreshing library: 16b5bc3b-5d3f-4290-bf56-fdf9589c076a
[Nest] 7  - 05/21/2024, 6:58:34 AM   DEBUG [ImmichMicroservices] [LibraryService] Found 0 asset(s) when crawling import paths /My_Photos/2002 pre-ADFA,/My_Photos/2003 - 2004 ADFA Years,/My_Photos/2005 - 2008 Vic Uni,/My_Photos/2008 - 2010 PSN - Motorbike Times AU,/My_Photos/2010 - 2012 EU,/My_Photos/2012 - Nth America,/My_Photos/2013 AU return,/My_Photos/2014 King Street,/My_Photos/2015-04-28 iPhone 3GS vidoes,/My_Photos/2015 King Street 2,/My_Photos/2016 King Street 3,/My_Photos/2016 Magnolia 10 Bowen Street,/My_Photos/2017 Magnolia 10 Bowen Street,/My_Photos/2018 Magnolia,/My_Photos/2019 Magnolia 10 Bowen St,/My_Photos/2020 Magnolia Isla,/My_Photos/2021 Magnolia Isla big sister - Lewis,/My_Photos/2022 Magnolia Isla Lewis renos continue,/My_Photos/2023 Magnolia Isla Lewis Hilux Camping,/My_Photos/M8 - 28OCT18/,/My_Photos/iphone 19MAR2013/,/My_Photos/Aliboo_Pixel3_Dec19-May23/,/My_Photos/Aliboo_HTCM7_Jul13-May16,/My_Photos/Alison's Photos/
[Nest] 7  - 05/21/2024, 6:58:34 AM     LOG [ImmichMicroservices] [LibraryService] Refreshing library: 16b5bc3b-5d3f-4290-bf56-fdf9589c076a
[Nest] 7  - 05/21/2024, 6:58:34 AM VERBOSE [ImmichMicroservices] [LibraryService] Crawled asset paths paginated
[Nest] 7  - 05/21/2024, 6:58:39 AM VERBOSE [ImmichMicroservices] [LibraryService] Crawled assets have been checked for online/offline status

FYI; my docker mount is (for reference above);
- /media/Shed4/My Photos/:/My_Photos:ro

I just completed the following test;

1. Delete 1 import path from the original External Library.
2. Scanned - nothing.
3. Added it back in - green tick and validated.
4. Scanned - nothing, again zero assets found. Re-added import path can be seen in the import path list above.
5. Force Scanned - nothing, zero assets found. Re-added import path can be seen in the import path list above.

Is it the import paths (and directories themselves) have spaces in them?

I also chmod'd everything in my external library to be 774.

[etnoy]

We did change the glob library a few months ago, the timing sounds about right.

Maybe you can use pgadmin to check the library table in the db.

[matticas]

That was 105.0 - I did read that, this was broken before then. I'm not using globs anyway in the imports, only in the exclusion paths.

[etnoy]

The glob library is used to crawl paths, it's been in the external library code since the start. It's always used, no matter if you glob paths or not

[etnoy]

Gut feeling right now is that we don't correctly handle spaces in import paths. You say it works when doing the test import folder without spaces, right?

[matticas]

Not entirely.
Previous to "asset offline" issues; all import paths had spaces, all imported to the original External Library.
At some point, they all went offline (not sure what version exactly when I upgraded each time).

Test; /My_Photos/test_import/ works.
Test2; /My_Photos/import path one with spaces/ works on a new External Library. Exact same path doesn't on the original External Library.

I've installed a PGadmin container, but i've never used a postgres database, i'm still learning how to connect to the immich postgres docker volume.

[matticas]

Hi etony;

I have pgadmin running and connected to the immich database, however I do not know what I am looking for within it to verify data paths?

Do you have information you could share so I can start searching and trying to validate?

Thanks

[etnoy]

Awesome. You said your had one working library and one that doesn't? Can you check the library table and compare the two?

[matticas]

I will try, I have never used a database or pgadmin, so far I've only connected, I haven't discovered how to display the tables. Edit; Ok, I have found the tables, and can view them. Give me some time to roam about!

[matticas]

Hi @etnoy

So I have discovered;

Table: libraries :: ImportPaths

Any import paths with spaces have " " around them; eg;
{"/docker_container_location/import path with space/","docker_container_location/import path with space2/"}

And import paths without spaces, have no "" around them; eg;
{/docker_container_location/import_test}

Table: assets :: originalPath

values in this column do not have "" around their paths, which the paths include spaces. eg;

'/docker_container_location/import path with space/file.jpg'

However, I have two test libraries, with and without spaces, that work. And the large original External Library that isn't working (with spaces).

Can also confirm, I have assets that are offline, that have no spaces in their import paths or file paths.

What else should I look for?

[matticas]

Hi @alextran1502 @etnoy

From the post above, if I modify the database ImportPath values, and it doesn't fix it, can I revert them back without isssues?

I'm at a loss for what to do to fix (without rebuilding), but I've never worked with databases, and I don't understand enough of the backend of the Immich Microservices to understand the mechanics of the "asset is offline" part.

Any hints or leads to chase to resolve would be much apricated so I can try and find the route cause of the issue.

thanks

[etnoy]

You can modify the import path values in the db, no worries.

@jrasm91, do you have any ideas?

[matticas]

You can modify the import path values in the db, no worries.

@jrasm91, do you have any ideas?

Hi @jrasm91 did you have a chance to look at this?

[figadore]

I am having a similar issue. In my case, I did a pg_dumpall from 1.98.2 and then restored to a new container (first to v1.105.1, then to 1.98.2 to reduce the number of variables), and all files have the "This asset is offline." message, even though the server and microservices containers can read those paths. However, there are a handful of iphone videos (where the path has no spaces) that still play, even though the offline error is there and there is no thumbnail. Not all videos play, I can't find anything else to distinguish those that will play and those that won't, and it may just be coincidence/red herring that the ones that do don't have a space in the path. I have tried with and without the pgdata docker volume. I currently have both the working and the broken containers running, what should I look for or test?

Apologies if this is not the same issue.

Edit: I'm running a long force-rescan of all libraries, which is definitely working hard. I'm not sure whether this step is recommended after a restore, but I'll update here if it makes things go back to normal

Edit edit: the force re-scan appears to have fixed it, perhaps this was unrelated. Sorry for the noise.

[matticas]

Yeah my Force Re-Scan doesn't detect any files.

I can even change the database offline boolean, and the Microservices resets it to offline again. Definitely a bug in my opinion.

[jenda69]

Hmm, I have a similar issue, but not all files are offline. Only like 10-15 files (from ~60000) are often marked as offline when using Scan New Library Files.

I just run the scan 3 times, first time I got 13 files marked as offline, second time no files offline (correct), third time the 13 same files again. But I remember a week ago there were at least some different files.

I think this never happened when using Re-scan All Library Files option. This scan takes bit longer and I think that's what I always successfully used to fix my issue.

[matticas]

Hmm, I have a similar issue, but not all files are offline. Only like 10-15 files (from ~60000) are often marked as offline when using Scan New Library Files.

I just run the scan 3 times, first time I got 13 files marked as offline, second time no files offline (correct), third time the 13 same files again. But I remember a week ago there were at least some different files.

I think this never happened when using Re-scan All Library Files option. This scan takes bit longer and I think that's what I always successfully used to fix my issue.

I'm just getting everything in my large external library as Offline, and the same folders in Smaller External Libraries as ok, spaces or no spaces.

I can't see a pattern, i've even been digging around within the Database itself looking for anything, but can't find it.

My Re-scans, all of them, no detect the files at all; "0 assets found when crawling......."

[rvoosterhout]

Maybe my solution will help, I noticed that my offline files had a different path than the path set in my external library.

#4925 (comment)

[matticas]

Maybe my solution will help, I noticed that my offline files had a different path than the path set in my external library.

#4925 (comment)

Read your reply, completely unrelated.

[matticas]

#6845 (reply in thread)

For those searching; please see above link.

Issue was found to be Single Quotation Marks in the directories (and sub-directories) of one particular Import Path.

The entire external library consisted of 24 import paths, and only one had Single Quotation Marks in the parent import path and multiple sub-directories.

The Import Path itself Validated fine with the validation button. But once saved and the External Library scanned, it rendered all assets in the entire external library (all 24 import paths) as Offline; "Found 0 asset(s) when crawling..."

The original external library was import in full around the 1.9xx.x days, not sure what exact release.