[BUG] All images uploaded are corrupt with CIFS mount

[valdearg]

The bug

I've previously been using Immich in the background and occasionally come to it to look. Recently I noticed that all images were coming up as broken links.

Today I've finally had some time to look into the issue and I'm not sure what's actually happening.

Only one image seems to be loading, all others (video and image) seem to be broken links.

In the JSON API response I receive what appears to be normal information:

{"id":"c62f37a2-bf5f-4dce-9170-a82f9fb7c6e9","type":"IMAGE","thumbhash":null,"localDateTime":"2023-10-20T16:38:47.000Z","resized":false,"duration":"0:00:00.00000","livePhotoVideoId":null,"hasMetadata":true,"deviceAssetId":"1000019644","ownerId":"d759d497-891f-4eaa-9b1c-6ae480fc2e1f","owner":{"id":"d759d497-891f-4eaa-9b1c-6ae480fc2e1f","email":"redacted","firstName":"redacted","lastName":"redacted","storageLabel":"admin","externalPath":null,"profileImagePath":"","shouldChangePassword":true,"isAdmin":true,"createdAt":"2023-10-20T16:59:08.753Z","deletedAt":null,"updatedAt":"2023-10-20T16:59:08.753Z","oauthId":"","memoriesEnabled":true},"deviceId":"acfe901afa96dcef8021b1da06540b63770c6a92008e7ce21ade432116fe972b","libraryId":"01083507-192b-4c71-b6ce-9a1e691c26be","originalPath":"upload/library/admin/2023/2023-10-20/20231020_173847.jpg","originalFileName":"20231020_173847","fileCreatedAt":"2023-10-20T16:38:47.000Z","fileModifiedAt":"2023-10-20T16:38:48.000Z","updatedAt":"2023-10-20T17:13:43.804Z","isFavorite":false,"isArchived":false,"isTrashed":false,"exifInfo":{"make":null,"model":null,"exifImageWidth":null,"exifImageHeight":null,"fileSizeInByte":65536,"orientation":null,"dateTimeOriginal":"2023-10-20T16:38:47.000Z","modifyDate":"2023-10-20T16:38:48.000Z","timeZone":null,"lensModel":null,"fNumber":null,"focalLength":null,"iso":null,"exposureTime":null,"latitude":null,"longitude":null,"city":null,"state":null,"country":null,"description":"","projectionType":null},"tags":[],"people":[],"checksum":"18mwWpVlutDtavqMTQ/yKkifbSQ=","isExternal":false,"isOffline":false,"isReadOnly":false}

The originalPath refers to: upload/library/admin/2023/2023-10-20/20231020_173847.jpg

This is mount via CIFS network storage which seems to be fine for other docker containers running on the same host.

When I check these items, they cannot be opened. They are all exactly 64KB in file size.

See below for one of the files:

I've done the following so far:

1. Removed the docker containers.
2. Removed the storage mounts.
3. Moved the physical storage, mounted within the docker container.
4. Downloaded the latest docker-compose.yml
5. Installed Immich clean.
6. Removed the Android app and re-installed.
7. Synced the files up.
8. Allowed the background processes to complete.
9. Files corrupted?
10. Added items through the web interface, also corrupt.

I can't seem to find anything obvious in any of the container logs.

The website is available using nginx-proxy-manager.

The images are fine in the Immich app when local:

Files seem to be loading in the Gallery app on the phone.

The OS that Immich Server is running on

Ubuntu Server 22.04.2 LTS

Version of Immich Server

v1.82.1

Version of Immich Mobile App

1.82.0 build.106

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

###################################################################################
# Database
###################################################################################

# NOTE: The following four database variables support Docker secrets by adding a *_FILE suffix to the variable name
# See the docker-compose documentation on secrets for additional details: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=<redacted>
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# REDIS_URL will be used to pass custom options to ioredis.
# Example for Sentinel
# {"sentinels":[{"host":"redis-sentinel-node-0","port":26379},{"host":"redis-sentinel-node-1","port":26379},{"host":"redis-sentinel-node-2","port":26379}],"name":"redis-sentinel"}
# REDIS_URL=ioredis://eyJzZW50aW5lbHMiOlt7Imhvc3QiOiJyZWRpcy1zZW50aW5lbDEiLCJwb3J0IjoyNjM3OX0seyJob3N0IjoicmVkaXMtc2VudGluZWwyIiwicG9ydCI6MjYzNzl9XSwibmFtZSI6Im15bWFzdGVyIn0=

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_USERNAME=
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/media/Rin/Media/Pictures/Immich


###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=<redacted>
# TYPESENSE_ENABLED=false
# TYPESENSE_URL uses base64 encoding for the nodes json.
# Example JSON that was used:
# [
#      { 'host': 'typesense-1.example.net', 'port': '443', 'protocol': 'https' },
#      { 'host': 'typesense-2.example.net', 'port': '443', 'protocol': 'https' },
#      { 'host': 'typesense-3.example.net', 'port': '443', 'protocol': 'https' },
#  ]
# TYPESENSE_URL=ha://WwogICAgeyAnaG9zdCc6ICd0eXBlc2Vuc2UtMS5leGFtcGxlLm5ldCcsICdwb3J0JzogJzQ0MycsICdwcm90b2NvbCc6ICdodHRwcycgfSwKICAgIHsgJ2hvc3QnOiAndHlwZXNlbnNlLTIuZXhhbXBsZS5uZXQnLCAncG9ydCc6ICc0NDMnLCAncHJvdG9jb2wnOiAnaHR0cHMnIH0sCiAgICB7ICdob3N0JzogJ3R5cGVzZW5zZS0zL>

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

Reproduction steps

6. Removed the Android app and re-installed.
7. Synced the files up.
8. Allowed the background processes to complete.
9. Files corrupted?
10. Added items through the web interface, also corrupt.

Additional information

No response

[jrasm91]

The broken link icon means no thumbnails have been generated. Make sure the microservices container is up and running.

[valdearg]

Looks like the microservices container is up and running, a lot of errors about the files themselves.

[Nest] 8  - 10/21/2023, 12:05:49 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 10/21/2023, 12:05:49 AM   ERROR [JobService] Object:
{
  "id": "3801495a-2c4b-4aee-82f3-e93a8f7a9496"
}
[Nest] 8  - 10/21/2023, 12:05:49 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8  - 10/21/2023, 12:05:49 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 10/21/2023, 12:05:49 AM   ERROR [JobService] Object:
{
  "id": "b17714ab-6974-4bc7-8022-5ac942237a04"
}
[Nest] 8  - 10/21/2023, 12:05:49 AM    WARN [MediaService] Skipped thumbnail generation for asset f0fbd532-6951-4e04-b8eb-31046002b270: no video streams found
[Nest] 8  - 10/21/2023, 12:05:49 AM    WARN [MediaService] Skipped thumbnail generation for asset d3e0c7f4-4d1c-49a9-a5ae-d9a04d52efa0: no video streams found
[Nest] 8  - 10/21/2023, 12:05:49 AM    WARN [MediaService] Skipped thumbnail generation for asset 6b01c84c-bfb9-4918-a610-2f8904b1bd96: no video streams found
[Nest] 8  - 10/21/2023, 12:05:49 AM    WARN [MediaService] Skipped thumbnail generation for asset 3527950c-f4df-4c26-833f-b79efa94b831: no video streams found
[Nest] 8  - 10/21/2023, 12:05:49 AM    WARN [MediaService] Skipped thumbnail generation for asset 77c8279c-1475-4f98-b891-9c2603bc8318: no video streams found
[Nest] 8  - 10/21/2023, 7:51:49 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8  - 10/21/2023, 7:51:49 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 10/21/2023, 7:51:49 AM   ERROR [JobService] Object:
{
  "id": "341cb599-824b-42ea-92ba-905d8a60a2ba",
  "source": "upload"
}

[xinmans]

same error，how to fix？

[alextran1502]

We are having an issue with CIFS mount with the new version of Node.js that is on the server. For context, here is the link nodejs/node#50061

You can add UV_USE_IO_URING=0 to the .env file for the workaround. We are monitoring to see when this issue will be fix upstream

[CranialRelics]

I was able to test this and it seems to work for new images. Are there any recommendations for reprocessing the images that were impacted by this? I deleted a few from the web interface and they seem to disappear on the mobile app and I'm not sure how best to approach reprocessing.

[alextran1502]

@CranialRelics, remember also to remove them from the trash page, then you can reupload them :)

[valdearg]

Thank you! That's precisely the issue. I've got it to re-upload everything just for piece of mind.