[BUG] Mobile background backup does not work with self signed certificate

[ktm-91]

The bug

When I take a picture on my phone and wait 5 seconds, instead of finding that photo uploaded to Immich server, I found 3 "SSL handshake failure" logs on my reverse proxy. Every other feature works as expected, through the reverse proxy, in HTTPS, including the foreground backup when I open the app.
I'm using HAProxy on pfSense with a self-signed certificate.

The OS that Immich Server is running on

Debian 12 x64 + Docker

Version of Immich Server

1.90.1

Version of Immich Mobile App

1.90.0 build.114

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

Not relevant (anyway, the default one)

Your .env content

Not relevant (anyway, the default one)

Reproduction steps

1. take a picture with the phone
2. wait 5 seconds without opening Immich app
3. the picture is not getting uploaded automatically in the background, instead the connection attempts of the app results in 3 "SSL handshake failure" logs (3 every picture that the app tries to upload in the background)

Additional information

The feature works correctly if I use the same reverse proxy without SSL Offloading and if I point directly to Immich server.

[ktm-91]

Same issue with latest Android app version 1.91.0

[ktm-91]

Am I the only one with this issue? Still present in the latest version on the app

[harshitandro]

Having similar issue with 1.91.4 version android app and server. Haproxy as reverse proxy.

[sivel27]

Same on v1.92.0 android.

[wociscz]

Same with 1.93.3 - created another issue before I stumbled upon this (same) issue. Not working also with valid (not self-signed) certificate.

[jacob-horton]

Same on v1.93.3 android:

• Certificates installed on android
• "Allow self-signed SSL certificates" enabled
• Traefik reverse proxy for immich server
• Default immich docker configuration
• Issue doesn't occur if using HTTP instead of HTTPS (still going through reverse proxy)
• Issue doesn't occur if using IP/port of immich server directly

[ktm-91]

Yeah the bug is still there, and I also found that with the reverse proxy I cannot play any video of my libraries from the Android app. I find the same certificate errors in HAProxy logs whenever I try to start playing.

[xxTBxx]

Seeing this with android 1.94.1 build 121. Foreground backup works fine but background backup throw errors on self signed certificates.

Server is running in K3s with Traefik ingress in K3s, config is straight from Immich Helm charts.

Background Upload Android Steps

With android app minimized photo taken and collected logs after failure notification.

app log level shout
logs pulled in adb shell using logcat --pid=$(pidof -s app.alextran.immich)

Tested with foreground upload setting both enabled and disabled.

Logs

02-07 20:15:27.716 16377 25795 I flutter : Disconnect to Websocket Connection
02-07 20:15:39.618 16377 29665 D BackupWorker: enqueueBackupWorker: BackupWorker enqueued
02-07 20:15:39.740 16377 16377 D BackupWorker: startWork
02-07 20:15:39.744 16377 16453 I WM-Processor: Moving WorkSpec (fa0dd64a-45b3-485b-a9ca-dffe984c6fab) to the foreground
02-07 20:15:39.815 16377 16377 I WM-SystemFgDispatcher: Started foreground service Intent { act=ACTION_START_FOREGROUND cmp=app.alextran.immich/androidx.work.impl.foreground.SystemForegroundService (has extras) }
02-07 20:15:39.815 16377 16377 D FlutterGeolocator: Flutter engine connected. Connected engine count 2
02-07 20:15:40.009 16377  3185 I flutter : Error [getDeviceBackupAsset] ApiException 400: TLS/SSL communication failed: GET /asset/device/5150272af11c21dfc7a0318d53261558bb76391675299b85103d46890f3683ed (Inner exception: HandshakeException: Handshake error in client (OS Error: 
02-07 20:15:40.009 16377  3185 I flutter : 	CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393)))
02-07 20:15:40.009 16377  3185 I flutter : 
02-07 20:15:40.009 16377  3185 I flutter : #0      _SecureFilterImpl._handshake (dart:io-patch/secure_socket_patch.dart:99)
02-07 20:15:40.009 16377  3185 I flutter : #1      _SecureFilterImpl.handshake (dart:io-patch/secure_socket_patch.dart:143)
02-07 20:15:40.009 16377  3185 I flutter : #2      _RawSecureSocket._secureHandshake (dart:io/secure_socket.dart:920)
02-07 20:15:40.009 16377  3185 I flutter : #3      _RawSecureSocket._tryFilter (dart:io/secure_socket.dart:1049)
02-07 20:15:40.010 16377  3185 I flutter : <asynchronous suspension>
02-07 20:15:40.010 16377  3185 I flutter : 
02-07 20:15:40.012 16377 16437 I WM-Processor: Moving WorkSpec (fa0dd64a-45b3-485b-a9ca-dffe984c6fab) to the foreground
02-07 20:15:40.037 16377  3185 I flutter : ERROR backupAsset: HandshakeException: Handshake error in client (OS Error: 
02-07 20:15:40.037 16377  3185 I flutter : 	CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393))
02-07 20:15:40.067 16377  3185 I flutter : ERROR backupAsset: HandshakeException: Handshake error in client (OS Error: 
02-07 20:15:40.067 16377  3185 I flutter : 	CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393))
02-07 20:15:40.073 16377 16377 D FlutterGeolocator: Flutter engine disconnected. Connected engine count 1
02-07 20:15:40.073 16377 16377 E FlutterGeolocator: Geolocator position updates stopped
02-07 20:15:40.073 16377 16377 E FlutterGeolocator: There is still another flutter engine connected, not stopping location service
02-07 20:15:40.106 16377 16377 D BackupWorker: stopEngine result=Success {mOutputData=Data {}}
02-07 20:15:40.107 16377 16397 I WM-WorkerWrapper: Worker result SUCCESS for Work [ id=fa0dd64a-45b3-485b-a9ca-dffe984c6fab, tags={ app.alextran.immich.BackupWorker } ]
02-07 20:15:40.110 16377 16377 I WM-SystemFgDispatcher: Stopping foreground service

Foreground Upload Logs

Given that the images have not been uploaded at this stage, if the app is opened and the foreground upload setting is enabled the pictures are successfully uploaded. Logs for the successful foreground upload are as follows:

02-07 20:46:46.739 16377 25795 I flutter : [APP STATE] hidden
02-07 20:46:46.739 16377 25795 I flutter : [APP STATE] inactive
02-07 20:46:46.805 16377 25795 I flutter : [APP STATE] resumed
02-07 20:46:46.805 16377 25795 I flutter : Attempting to connect to websocket
02-07 20:46:46.817 16377 16377 D ContentObserverWorker: enabled ContentObserverWorker
02-07 20:46:46.817 16377 16377 D ContentObserverWorker: enabled ContentObserverWorker
02-07 20:46:46.824 16377 25795 I flutter : Start backup process
02-07 20:46:46.843 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:46.865 16377 25795 I flutter : Established Websocket Connection
02-07 20:46:46.889 16377 25795 I flutter : refreshRemoteAssets full took 83ms
02-07 20:46:47.059 16377 25795 I flutter : refreshDeviceAlbums took 170ms
02-07 20:46:47.060 16377 25795 I flutter : newRemote: true, newLocal: true
02-07 20:46:47.081 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.100 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.186 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.203 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.235 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.253 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.269 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.286 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.303 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.319 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.336 16377 25796 E alextran.immich: PIXEL: ioctl err: 1
02-07 20:46:47.653 16377 16513 D ExifInterface: No image meets the size requirements of a thumbnail image.
02-07 20:46:47.869 16377 25795 I flutter : _getBackupAlbumsInfo takes 1044ms
02-07 20:46:55.102 16377 25795 I flutter : [APP STATE] inactive
02-07 20:46:55.242 16377 25795 I flutter : [APP STATE] hidden

[ktm-91]

Do you also have the same issue trying to play videos from Android app through reverse proxy?

[Sammy1Am]

I'm having the same issue with version 1.101. Also using Traefik proxy, but using Let's Encrypt signed certificates. Even though my certs aren't self-signed, I still had to check the "ignore self-signed" (or whatever) setting before I could even login.

Sounds SSL related somehow maybe, or maybe just a Traefik thing?

[ktm-91]

I'm having the same issue with version 1.101. Also using Traefik proxy, but using Let's Encrypt signed certificates. Even though my certs aren't self-signed, I still had to check the "ignore self-signed" (or whatever) setting before I could even login.

Sounds SSL related somehow maybe, or maybe just a Traefik thing?

It's not a Traefik issue, I'm using HAProxy with a Let's Encrypt certificate as well. I cannot make neither the background backup feature nor the video playback working on Android

[CommanderBubble]

i've got the same issue with 1.102.3, and as people have described, it seems to be a TLS issue

i spun up a new instance on a VM to test it wasn't something else i'd done as well
running a default install straight from the getting started, and then adding traefik as a reverse proxy

• ip:port works
• http works
• https (self-signed) fails in background

the logs don't show anything that looks like an error either logs.log

[sidamos]

I have the same issue on iOS and tested a lot of combinations with Apache and Caddy Reverse Proxy, external subdomain, direct connection, valid and invalid certs. The foreground operations always worked fine, issue is only with the background sync. Ignore invalid certs is ON in settings.

Background sync works:

• http, direct connection in local network
• https, external subdomain, Reverse Proxy (Caddy, but Apache should work too), valid cert from Let's Encrypt

Background sync works not:

• https, external subdomain, Reverse Proxy (Caddy or Apache), cert from base domain (= invalid for subdomain)

Basic Auth inside the URL also is not a problem, unless there are special characters in the password like "#". Does not work, even when encoded. Cannot login.

BTW, I cannot always change the "ignore invalid certs" setting. Sometimes it is disabled.

[jasonmhite]

Possibly related, I'm also having trouble with the background sync not working on Android. I'm using a certificate signed by my own root certificate authority, which I have also added to the Android root certificate store. It looks like Immich doesn't see this as a valid cert, perhaps related to the recent changes to how Google handles custom installed certificates and apps needing to opt in. Immich won't let me log in without checking the ignore self-signed certs, even though it is most definitely valid. I also get the same effects with background sync not working as other people described, so I think I'm getting treated the same way as a self signed certificate.