feat(ml): support multiple urls

docs/docs/guides/remote-machine-learning.md

@@ -1,18 +1,20 @@
 # Remote Machine Learning
 
-To alleviate [performance issues on low-memory systems](/docs/FAQ.mdx#why-is-immich-slow-on-low-memory-systems-like-the-raspberry-pi) like the Raspberry Pi, you may also host Immich's machine-learning container on a more powerful system (e.g. your laptop or desktop computer):
-
-- Set the URL in Machine Learning Settings on the Admin Settings page to point to the designated ML system, e.g. `http://workstation:3003`.
-- Copy the following `docker-compose.yml` to your ML system.
-  - If using [hardware acceleration](/docs/features/ml-hardware-acceleration), the [hwaccel.ml.yml](https://github.com/immich-app/immich/releases/latest/download/hwaccel.ml.yml) file also needs to be added
-- Start the container by running `docker compose up -d`.
+To alleviate [performance issues on low-memory systems](/docs/FAQ.mdx#why-is-immich-slow-on-low-memory-systems-like-the-raspberry-pi) like the Raspberry Pi, you may also host Immich's machine learning container on a more powerful system, such as your laptop or desktop computer. The server container will send requests containing the image preview to the remote machine learning container for processing. The machine learning container does not persist this data or associate it with a particular user.
 
 :::info
-Smart Search and Face Detection will use this feature, but Facial Recognition is handled in the server.
+Smart Search and Face Detection will use this feature, but Facial Recognition will not. This is because Facial Recognition uses the _outputs_ of these models that have already been saved to the database. As such, its processing is between the server container and the database.
 :::
 
 :::danger
-When using remote machine learning, the thumbnails are sent to the remote machine learning container. Use this option carefully when running this on a public computer or a paid processing cloud.
+Image previews are sent to the remote machine learning container. Use this option carefully when running this on a public computer or a paid processing cloud. Additionally, as an internal service, the machine learning container has no security measures whatsoever. Please be mindful of where it's deployed and who can access it.
+:::
+
+1. Ensure the remote server has Docker installed
+2. Copy the following `docker-compose.yml` to the remote server
+
+:::info
+If using hardware acceleration, the [hwaccel.ml.yml](https://github.com/immich-app/immich/releases/latest/download/hwaccel.ml.yml) file also needs to be added and the `docker-compose.yml` needs to be configured as described in the [hardware acceleration documentation](/docs/features/ml-hardware-acceleration)
 :::
 
 ```yaml
@@ -37,8 +39,26 @@ volumes:
   model-cache:
 ```
 
-Please note that version mismatches between both hosts may cause instabilities and bugs, so make sure to always perform updates together.
+3. Start the remote machine learning container by running `docker compose up -d`
+
+:::info
+Version mismatches between both hosts may cause bugs and instability, so remember to update this container as well when updating the local Immich instance.
+:::
+
+4. Navigate to the [Machine Learning Settings](https://my.immich.app/admin/system-settings?isOpen=machine-learning)
+5. Click _Add URL_
+6. Fill the new field with the URL to the remote machine learning container. In the simplest case where the server is on the same local network, this may use the server's private IP. For example, if the private IP of the remote server is 192.168.0.50, the resulting URL will be `http://192.168.0.50:3003`. More advanced configuration is left as an exercise to the reader
+
+## Forcing remote processing
+
+Adding a new URL to the settings is recommended over replacing the existing URL. This is because it will allow machine learning tasks to be processed successfully when the remote server is down by falling back to the local machine learning container. If you do not want machine learning tasks to be processed locally when the remote server is not available, you can instead replace the existing URL and only provide the remote container's URL. If doing this, you can remove the `immich-machine-learning` section of the local `docker-compose.yml` file to save resources, as this service will never be used.
+
+Do note that this will mean that Smart Search and Face Detection jobs will fail to be processed when the remote instance is not available. This in turn means that tasks dependent on these features—Duplicate Detection and Facial Recognition—will not run for affected assets. If this occurs, you must manually click the _Missing_ button next to Smart Search and Face Detection in the [Job Status](http://my.immich.app/admin/jobs-status) page for the jobs to be retried.
+
+## Load balancing
+
+While several URLs can be provided in the settings, they are tried sequentially; there is no attempt to distribute load across multiple containers. It is recommended to use a dedicated load balancer for such use-cases and specify it as the only URL. Among other things, it may enable the use of different APIs on the same server by running multiple containers with different configurations. For example, one might run an OpenVINO container in addition to a CUDA container, or run a standard release container to maximize both CPU and GPU utilization.
 
-:::caution
-As an internal service, the machine learning container has no security measures whatsoever. Please be mindful of where it's deployed and who can access it.
+:::tip
+The machine learning container can be shared among several Immich instances, regardless of the models a particular instance uses. However, using different models will lead to higher peak memory usage.
 :::

docs/docs/install/config-file.md

@@ -83,7 +83,7 @@ The default configuration looks like this:
   },
   "machineLearning": {
     "enabled": true,
-    "url": "http://immich-machine-learning:3003",
+    "url": ["http://immich-machine-learning:3003"],
     "clip": {
       "enabled": true,
       "modelName": "ViT-B-32__openai"

i18n/en.json

@@ -23,6 +23,7 @@
   "add_to": "Add to...",
   "add_to_album": "Add to album",
   "add_to_shared_album": "Add to shared album",
+  "add_url": "Add URL",
   "added_to_archive": "Added to archive",
   "added_to_favorites": "Added to favorites",
   "added_to_favorites_count": "Added {count, number} to favorites",
@@ -130,7 +131,7 @@
     "machine_learning_smart_search_description": "Search for images semantically using CLIP embeddings",
     "machine_learning_smart_search_enabled": "Enable smart search",
     "machine_learning_smart_search_enabled_description": "If disabled, images will not be encoded for smart search.",
-    "machine_learning_url_description": "URL of the machine learning server",
+    "machine_learning_url_description": "The URL of the machine learning server. If more than one URL is provided, each server will be attempted one-at-a-time until one responds successfully, in order from first to last.",
     "manage_concurrency": "Manage Concurrency",
     "manage_log_settings": "Manage log settings",
     "map_dark_style": "Dark style",
@@ -1041,6 +1042,7 @@
   "remove_from_album": "Remove from album",
   "remove_from_favorites": "Remove from favorites",
   "remove_from_shared_link": "Remove from shared link",
+  "remove_url": "Remove URL",
   "remove_user": "Remove user",
   "removed_api_key": "Removed API Key: {name}",
   "removed_from_archive": "Removed from archive",

open-api/immich-openapi-specs.json

@@ -11853,7 +11853,12 @@
             "$ref": "#/components/schemas/FacialRecognitionConfig"
           },
           "url": {
-            "type": "string"
+            "items": {
+              "format": "uri",
+              "type": "string"
+            },
+            "minItems": 1,
+            "type": "array"
           }
         },
         "required": [

open-api/typescript-sdk/src/fetch-client.ts

@@ -1177,7 +1177,7 @@ export type SystemConfigMachineLearningDto = {
     duplicateDetection: DuplicateDetectionConfig;
     enabled: boolean;
     facialRecognition: FacialRecognitionConfig;
-    url: string;
+    url: string[];
 };
 export type SystemConfigMapDto = {
     darkStyle: string;

server/src/config.ts

@@ -52,7 +52,7 @@ export interface SystemConfig {
   };
   machineLearning: {
     enabled: boolean;
-    url: string;
+    url: string[];
     clip: {
       enabled: boolean;
       modelName: string;
@@ -205,7 +205,7 @@ export const defaults = Object.freeze<SystemConfig>({
   },
   machineLearning: {
     enabled: process.env.IMMICH_MACHINE_LEARNING_ENABLED !== 'false',
-    url: process.env.IMMICH_MACHINE_LEARNING_URL || 'http://immich-machine-learning:3003',
+    url: [process.env.IMMICH_MACHINE_LEARNING_URL || 'http://immich-machine-learning:3003'],
     clip: {
       enabled: true,
       modelName: 'ViT-B-32__openai',

server/src/dtos/system-config.dto.ts

@@ -1,6 +1,7 @@
 import { ApiProperty } from '@nestjs/swagger';
-import { Type } from 'class-transformer';
+import { Transform, Type } from 'class-transformer';
 import {
+  ArrayMinSize,
   IsBoolean,
   IsEnum,
   IsInt,
@@ -269,9 +270,12 @@ class SystemConfigMachineLearningDto {
   @ValidateBoolean()
   enabled!: boolean;
 
-  @IsUrl({ require_tld: false, allow_underscores: true })
+  @IsUrl({ require_tld: false, allow_underscores: true }, { each: true })
+  @ArrayMinSize(1)
+  @Transform(({ value }) => (Array.isArray(value) ? value : [value]))
   @ValidateIf((dto) => dto.enabled)
-  url!: string;
+  @ApiProperty({ type: 'array', items: { type: 'string', format: 'uri' }, minItems: 1 })
+  url!: string[];
 
   @Type(() => CLIPConfig)
   @ValidateNested()

server/src/interfaces/machine-learning.interface.ts

@@ -51,7 +51,7 @@ export type DetectedFaces = { faces: Face[] } & VisualResponse;
 export type MachineLearningRequest = ClipVisualRequest | ClipTextualRequest | FacialRecognitionRequest;
 
 export interface IMachineLearningRepository {
-  encodeImage(url: string, imagePath: string, config: ModelOptions): Promise<number[]>;
-  encodeText(url: string, text: string, config: ModelOptions): Promise<number[]>;
-  detectFaces(url: string, imagePath: string, config: FaceDetectionOptions): Promise<DetectedFaces>;
+  encodeImage(url: string[], imagePath: string, config: ModelOptions): Promise<number[]>;
+  encodeText(url: string[], text: string, config: ModelOptions): Promise<number[]>;
+  detectFaces(url: string[], imagePath: string, config: FaceDetectionOptions): Promise<DetectedFaces>;
 }

server/src/repositories/machine-learning.repository.ts

@@ -1,6 +1,7 @@
-import { Injectable } from '@nestjs/common';
+import { Inject, Injectable } from '@nestjs/common';
 import { readFile } from 'node:fs/promises';
 import { CLIPConfig } from 'src/dtos/model-config.dto';
+import { ILoggerRepository } from 'src/interfaces/logger.interface';
 import {
   ClipTextualResponse,
   ClipVisualResponse,
@@ -13,26 +14,35 @@ import {
   ModelType,
 } from 'src/interfaces/machine-learning.interface';
 
-const errorPrefix = 'Machine learning request';
-
 @Injectable()
 export class MachineLearningRepository implements IMachineLearningRepository {
-  private async predict<T>(url: string, payload: ModelPayload, config: MachineLearningRequest): Promise<T> {
-    const formData = await this.getFormData(payload, config);
+  constructor(@Inject(ILoggerRepository) private logger: ILoggerRepository) {
+    this.logger.setContext(MachineLearningRepository.name);
+  }
 
-    const res = await fetch(new URL('/predict', url), { method: 'POST', body: formData }).catch(
-      (error: Error | any) => {
-        throw new Error(`${errorPrefix} to "${url}" failed with ${error?.cause || error}`);
-      },
-    );
+  private async predict<T>(urls: string[], payload: ModelPayload, config: MachineLearningRequest): Promise<T> {
+    const formData = await this.getFormData(payload, config);
+    for (const url of urls) {
+      try {
+        const response = await fetch(new URL('/predict', url), { method: 'POST', body: formData });
+        if (response.ok) {
+          return response.json();
+        }
 
-    if (res.status >= 400) {
-      throw new Error(`${errorPrefix} '${JSON.stringify(config)}' failed with status ${res.status}: ${res.statusText}`);
+        this.logger.warn(
+          `Machine learning request to "${url}" failed with status ${response.status}: ${response.statusText}`,
+        );
+      } catch (error: Error | unknown) {
+        this.logger.warn(
+          `Machine learning request to "${url}" failed: ${error instanceof Error ? error.message : error}`,
+        );
+      }
     }
-    return res.json();
+
+    throw new Error(`Machine learning request '${JSON.stringify(config)}' failed for all URLs`);
   }
 
-  async detectFaces(url: string, imagePath: string, { modelName, minScore }: FaceDetectionOptions) {
+  async detectFaces(url: string[], imagePath: string, { modelName, minScore }: FaceDetectionOptions) {
     const request = {
       [ModelTask.FACIAL_RECOGNITION]: {
         [ModelType.DETECTION]: { modelName, options: { minScore } },
@@ -47,13 +57,13 @@ export class MachineLearningRepository implements IMachineLearningRepository {
     };
   }
 
-  async encodeImage(url: string, imagePath: string, { modelName }: CLIPConfig) {
+  async encodeImage(url: string[], imagePath: string, { modelName }: CLIPConfig) {
     const request = { [ModelTask.SEARCH]: { [ModelType.VISUAL]: { modelName } } };
     const response = await this.predict<ClipVisualResponse>(url, { imagePath }, request);
     return response[ModelTask.SEARCH];
   }
 
-  async encodeText(url: string, text: string, { modelName }: CLIPConfig) {
+  async encodeText(url: string[], text: string, { modelName }: CLIPConfig) {
     const request = { [ModelTask.SEARCH]: { [ModelType.TEXTUAL]: { modelName } } };
     const response = await this.predict<ClipTextualResponse>(url, { text }, request);
     return response[ModelTask.SEARCH];

server/src/services/person.service.spec.ts

@@ -717,7 +717,7 @@ describe(PersonService.name, () => {
       assetMock.getByIds.mockResolvedValue([assetStub.image]);
       await sut.handleDetectFaces({ id: assetStub.image.id });
       expect(machineLearningMock.detectFaces).toHaveBeenCalledWith(
-        'http://immich-machine-learning:3003',
+        ['http://immich-machine-learning:3003'],
         '/uploads/user-id/thumbs/path.jpg',
         expect.objectContaining({ minScore: 0.7, modelName: 'buffalo_l' }),
       );

server/src/services/smart-info.service.spec.ts

@@ -289,7 +289,7 @@ describe(SmartInfoService.name, () => {
       expect(await sut.handleEncodeClip({ id: assetStub.image.id })).toEqual(JobStatus.SUCCESS);
 
       expect(machineLearningMock.encodeImage).toHaveBeenCalledWith(
-        'http://immich-machine-learning:3003',
+        ['http://immich-machine-learning:3003'],
         '/uploads/user-id/thumbs/path.jpg',
         expect.objectContaining({ modelName: 'ViT-B-32__openai' }),
       );
@@ -322,7 +322,7 @@ describe(SmartInfoService.name, () => {
 
       expect(databaseMock.wait).toHaveBeenCalledWith(512);
       expect(machineLearningMock.encodeImage).toHaveBeenCalledWith(
-        'http://immich-machine-learning:3003',
+        ['http://immich-machine-learning:3003'],
         '/uploads/user-id/thumbs/path.jpg',
         expect.objectContaining({ modelName: 'ViT-B-32__openai' }),
       );

server/src/services/system-config.service.spec.ts

@@ -85,7 +85,7 @@ const updatedConfig = Object.freeze<SystemConfig>({
   },
   machineLearning: {
     enabled: true,
-    url: 'http://immich-machine-learning:3003',
+    url: ['http://immich-machine-learning:3003'],
     clip: {
       enabled: true,
       modelName: 'ViT-B-32__openai',
@@ -329,11 +329,11 @@ describe(SystemConfigService.name, () => {
 
     it('should allow underscores in the machine learning url', async () => {
       configMock.getEnv.mockReturnValue(mockEnvData({ configFile: 'immich-config.json' }));
-      const partialConfig = { machineLearning: { url: 'immich_machine_learning' } };
+      const partialConfig = { machineLearning: { url: ['immich_machine_learning'] } };
       systemMock.readFile.mockResolvedValue(JSON.stringify(partialConfig));
 
       const config = await sut.getSystemConfig();
-      expect(config.machineLearning.url).toEqual('immich_machine_learning');
+      expect(config.machineLearning.url).toEqual(['immich_machine_learning']);
     });
 
     const externalDomainTests = [

web/src/lib/components/admin-page/settings/machine-learning-settings/machine-learning-settings.svelte

@@ -12,6 +12,9 @@
   import { t } from 'svelte-i18n';
   import FormatMessage from '$lib/components/i18n/format-message.svelte';
   import { SettingInputFieldType } from '$lib/constants';
+  import Button from '$lib/components/elements/buttons/button.svelte';
+  import CircleIconButton from '$lib/components/elements/buttons/circle-icon-button.svelte';
+  import { mdiMinusCircle } from '@mdi/js';
 
   interface Props {
     savedConfig: SystemConfigDto;
@@ -42,15 +45,36 @@
 
         <hr />
 
-        <SettingInputField
-          inputType={SettingInputFieldType.TEXT}
-          label={$t('url')}
-          description={$t('admin.machine_learning_url_description')}
-          bind:value={config.machineLearning.url}
-          required={true}
-          disabled={disabled || !config.machineLearning.enabled}
-          isEdited={config.machineLearning.url !== savedConfig.machineLearning.url}
-        />
+        {#each config.machineLearning.url as _, i}
+          {#snippet buttonSnippet()}
+            {#if config.machineLearning.url.length > 1}
+              <CircleIconButton
+                size="24"
+                class="ml-2"
+                padding="2"
+                color="red"
+                title=""
+                onclick={() => config.machineLearning.url.splice(i, 1)}
+                icon={mdiMinusCircle}
+              />
+            {/if}
+          {/snippet}
+
+          <SettingInputField
+            inputType={SettingInputFieldType.TEXT}
+            label={i === 0 ? $t('url') : undefined}
+            description={i === 0 ? $t('admin.machine_learning_url_description') : undefined}
+            bind:value={config.machineLearning.url[i]}
+            required={i === 0}
+            disabled={disabled || !config.machineLearning.enabled}
+            isEdited={i === 0 && !isEqual(config.machineLearning.url, savedConfig.machineLearning.url)}
+            {buttonSnippet}
+          />
+        {/each}
+
+        <Button class="mb-2" type="button" size="sm" onclick={() => config.machineLearning.url.splice(0, 0, '')}
+          >{$t('add_url')}</Button
+        >
       </div>
 
       <SettingAccordion