feat(web): deduplication UI #9540

alextran1502 · 2024-05-16T21:15:15Z

This PR implements

UI for deduplication view
Endpoint for resolving duplication

Utility Menu

Duplication UI - 1

Duplication UI - 2

Duplication UI - 3

Duplication UI - 4

Duplication Detection Settings

cloudflare-workers-and-pages · 2024-05-16T21:15:17Z

Deploying immich with Cloudflare Pages

Latest commit:	`e607958`
Status:	✅ Deploy successful!
Preview URL:	https://9623f74a.immich.pages.dev
Branch Preview URL:	https://feat-web-duplication-detecti.immich.pages.dev

View logs

…lication-detection-ui

NicholasFlamy · 2024-05-17T16:56:20Z

Backend for this UI (commenting so that the reference is here):
#8228

…lication-detection-ui

NicholasFlamy · 2024-05-17T19:20:34Z

I tested a fix to this problem:
https://discord.com/channels/979116623879368755/1237891913059930142/1241068695401267230

It's about Duplicate Detection not disabling when Machine Learning is enabled but Smart Search is disabled. Since Duplicate Detection relies on CLIP embeddings it should probably be disabled when Smart Search is disabled.

Should I PR this branch or main? This change is not specific to the UI, it's one small change to the server.

NicholasFlamy · 2024-05-17T19:29:15Z

I have a version for main and a version for this PR's branch but it applies to the existing implementation so it should probably be added to main:
https://github.com/NicholasFlamy/immich/tree/dupe-det-disable-main-branch

alextran1502 · 2024-05-17T19:38:42Z

I tested a fix to this problem: https://discord.com/channels/979116623879368755/1237891913059930142/1241068695401267230

It's about Duplicate Detection not disabling when Machine Learning is enabled but Smart Search is disabled. Since Duplicate Detection relies on CLIP embeddings it should probably be disabled when Smart Search is disabled.

Should I PR this branch or main? This change is not specific to the UI, it's one small change to the server.

@NicholasFlamy Please open a PR to main

NicholasFlamy · 2024-05-17T20:39:33Z

The change in #9565 should be drop in and shouldn't conflict with anything in this PR branch.

NicholasFlamy · 2024-05-18T14:26:27Z

If this works across libraries it'd be could to display which library the photo is from (so if they have one photo in the upload library and one in an external library, it should show that.

alextran1502 · 2024-05-18T15:34:38Z

@NicholasFlamy this will work with per account, not across instances.

NicholasFlamy · 2024-05-18T15:49:59Z

@NicholasFlamy this will work with per account, not across instances.

Sorry I didn't phrase it well. I confirmed with Mert that pictures from User A's external library can show up as duplicates of pictures in User A's upload library on the same instance of immich.

My suggestion is when User A goes to the duplicates page, the picture should have a label that tells whether the picture is in User A's external library or User A's upload library.

alextran1502 · 2024-05-18T15:59:43Z

@NicholasFlamy we will have to rework the permission/query then. Let move our discussions back to Discord since the pr discussion should be used for code review and code related

…lication-detection-ui

danieldietzler · 2024-05-22T19:26:19Z

...ib/components/admin-page/settings/machine-learning-settings/machine-learning-settings.svelte

+            step="0.01"
+            min={0.001}


@alextran1502 you want to stick with 0.01 for now?

web/src/lib/components/utilities-page/duplicates/duplicates-compare-control.svelte

web/src/routes/(user)/utilities/[duplicates]/[[photos=photos]]/[[assetId=id]]/+page.svelte

web/src/lib/components/utilities-page/duplicates/duplicates-compare-control.svelte

jrasm91

Noice

jrasm91 · 2024-05-23T03:31:53Z

server/src/dtos/duplicate.dto.ts

@@ -6,6 +7,11 @@ export class DuplicateResponseDto {
  assets!: AssetResponseDto[];
 }

+export class ResolveDuplicatesDto {
+  @IsNotEmpty()
+  assetIds!: string[];


I think you need @ValidateUUID({ each: true }) here

jrasm91 · 2024-05-23T03:32:41Z

...ib/components/admin-page/settings/machine-learning-settings/machine-learning-settings.svelte

+              !config.machineLearning.enabled ||
+              !config.machineLearning.clip.enabled ||
+              !config.machineLearning.duplicateDetection.enabled}


might be able to just use a feature flag here instead. $featureFlag.duplicateDetection

@NicholasFlamy What was the reason you didn't use the feature flag? I remember you have a PR put in for this, correct?

I added the feature flag to the original backend PR. I didn't think to use the feature flag here (mainly I forgot about this part when I added that flag). So I think a feature flag could replace all of that.

Suggested change

!config.machineLearning.enabled ||

!config.machineLearning.clip.enabled ||

!config.machineLearning.duplicateDetection.enabled}

$featureFlag.duplicateDetection

I haven't tested this, either I'll make the change on my test instance or if it looks good you could commit it and I'll test it, but I won't get to testing either option until tomorrow.

jrasm91 · 2024-05-23T03:34:08Z

web/src/routes/(user)/utilities/[duplicates]/[[photos=photos]]/[[assetId=id]]/+page.svelte

+      {/key}
+    {:else}
+      <div class="text-center text-lg dark:text-white flex place-items-center place-content-center">
+        No duplication was found on the instance


Suggested change

No duplication was found on the instance

No duplicates were found.

And probably a <p> tag instead of a div.

danieldietzler

Great work! :)

mertalev · 2024-05-23T19:31:45Z

...ib/components/admin-page/settings/machine-learning-settings/machine-learning-settings.svelte

-              !config.machineLearning.enabled ||
-              !config.machineLearning.clip.enabled ||
-              !config.machineLearning.duplicateDetection.enabled}
+            disabled={disabled || $featureFlags.duplicateDetection}


Shouldn't this be !$featureFlags.duplicateDetection?

Most certainly

rithask · 2024-05-24T09:11:36Z

an undo option and an option to skip checking duplicate videos would be nice

mertalev · 2024-05-24T15:12:13Z

It could be nice QoL for the notification to have an Undo button (also for archiving, trashing, etc.). The app could just wait for a certain time before sending the requests and cancel if you click Undo before the notification goes away. But I think this isn't strictly a duplicate-related request.

When you mention videos, I just realized that the duplicate detection will consider an image and a video with a similar thumbnail as duplicates. That doesn't make sense: it should check for assets of the same type.

There is also a possible issue where the thumbnail for videos can be black and spuriously match against other videos. This is a general issue with video thumbnails that we should fix.

But assuming those issues are addressed, would you still like to disable detection for videos? If so, could you elaborate on why?

ThomasConrad · 2024-06-07T19:06:12Z

Will there be an option to group duplicates automatically like google photos does instead of deleting them?

alextran1502 · 2024-06-07T19:24:14Z

@ThomasConrad maybe the next steps for this UI work

Amerlander · 2024-06-22T11:15:58Z

I want to use this feature for my external library. Sometimes there are two dublicate images, that just are located in different folders. So it would be great to also get more meta informations available about the images, like the full path.

Maybe also if there are any faces tagged, available metatags of the image etc.

And it would be great, if I dont have to make a decission to see other dubliactes. Going back and fourth trough the dublicates or just skipping would be helpfull to decide later on some matches.

NicholasFlamy · 2024-06-22T13:51:40Z

I want to use this feature for my external library.

That already works. As long as the libraries are on the same user in immich.

And it would be great, if I dont have to make a decission to see other dubliactes. Going back and fourth trough the dublicates or just skipping would be helpfull to decide later on some matches.

I agree, I suggested this, it's a maybe for the future.

feat(web): deduplication UI

2649e48

fix: missing authenticated decorator

a36e7fb

zackpollard added the 🖥️web label May 16, 2024

alextran1502 mentioned this pull request May 17, 2024

Duplicate detection UI test #9537

Closed

alextran1502 added 6 commits May 17, 2024 08:44

Merge branch 'main' of github.com:immich-app/immich into feat/web-dup…

9645b52

…lication-detection-ui

merge main

3647ebb

utilities menu

6713115

Merge branch 'main' of github.com:immich-app/immich into feat/web-dup…

ecc3469

…lication-detection-ui

Merge branch 'main' of github.com:immich-app/immich into feat/web-dup…

aa92329

…lication-detection-ui

router

2bd8c94

Merge branch 'main' of github.com:immich-app/immich into feat/web-dup…

e330b7f

…lication-detection-ui

NicholasFlamy mentioned this pull request May 17, 2024

fix(server): Disable duplicate detection when smart search disabled #9565

Merged

naming convention

aa62282

get assets

5341422

alextran1502 added 6 commits May 18, 2024 13:19

Merge branch 'main' of github.com:immich-app/immich into feat/web-dup…

02b6275

…lication-detection-ui

openapi

74c4987

Merge branch 'main' of github.com:immich-app/immich into feat/web-dup…

d683b63

…lication-detection-ui

update

20f8c66

building out controller

d168b13

action button

86425ef

alextran1502 added 5 commits May 22, 2024 10:55

merge main

f8df214

using s

ac940d8

controller signature

bc6b055

linting

91e577e

merge main

2c161a0

alextran1502 requested review from jrasm91, mertalev and danieldietzler May 22, 2024 17:54

remove generated file from pr

aea6909

danieldietzler reviewed May 22, 2024

View reviewed changes

mertalev reviewed May 22, 2024

View reviewed changes

web/src/lib/components/utilities-page/duplicates/duplicates-compare-control.svelte Outdated Show resolved Hide resolved

alextran1502 added 2 commits May 22, 2024 16:27

refactor

5bca35a

fix sort issue

c723db6

danieldietzler reviewed May 22, 2024

View reviewed changes

web/src/lib/components/utilities-page/duplicates/duplicates-compare-control.svelte Outdated Show resolved Hide resolved

alextran1502 added 3 commits May 22, 2024 16:57

pr feedback

b870524

pr feedback done

d907a93

remove ignore openapi file

bb48d3a

jrasm91 approved these changes May 23, 2024

View reviewed changes

alextran1502 added 2 commits May 22, 2024 22:34

remove ignore openapi file

1a9d21b

pr feedback

e607958

danieldietzler approved these changes May 23, 2024

View reviewed changes

alextran1502 merged commit 57d94bc into main May 23, 2024
24 checks passed

alextran1502 deleted the feat/web-duplication-detection-ui branch May 23, 2024 17:57

mertalev reviewed May 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(web): deduplication UI #9540

feat(web): deduplication UI #9540

alextran1502 commented May 16, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented May 16, 2024 •

edited

Loading

NicholasFlamy commented May 17, 2024

NicholasFlamy commented May 17, 2024 •

edited

Loading

NicholasFlamy commented May 17, 2024

alextran1502 commented May 17, 2024

NicholasFlamy commented May 17, 2024

NicholasFlamy commented May 18, 2024

alextran1502 commented May 18, 2024

NicholasFlamy commented May 18, 2024 •

edited

Loading

alextran1502 commented May 18, 2024

danieldietzler May 22, 2024

jrasm91 left a comment

jrasm91 May 23, 2024

jrasm91 May 23, 2024

alextran1502 May 23, 2024

NicholasFlamy May 23, 2024

jrasm91 May 23, 2024

jrasm91 May 23, 2024

danieldietzler left a comment

mertalev May 23, 2024

danieldietzler May 23, 2024

rithask commented May 24, 2024

mertalev commented May 24, 2024

ThomasConrad commented Jun 7, 2024

alextran1502 commented Jun 7, 2024

Amerlander commented Jun 22, 2024 •

edited

Loading

NicholasFlamy commented Jun 22, 2024

	No duplication was found on the instance
	No duplicates were found.

feat(web): deduplication UI #9540

feat(web): deduplication UI #9540

Conversation

alextran1502 commented May 16, 2024 • edited Loading

Utility Menu

Duplication UI - 1

Duplication UI - 2

Duplication UI - 3

Duplication UI - 4

Duplication Detection Settings

cloudflare-workers-and-pages bot commented May 16, 2024 • edited Loading

Deploying immich with Cloudflare Pages

NicholasFlamy commented May 17, 2024

NicholasFlamy commented May 17, 2024 • edited Loading

NicholasFlamy commented May 17, 2024

alextran1502 commented May 17, 2024

NicholasFlamy commented May 17, 2024

NicholasFlamy commented May 18, 2024

alextran1502 commented May 18, 2024

NicholasFlamy commented May 18, 2024 • edited Loading

alextran1502 commented May 18, 2024

Choose a reason for hiding this comment

jrasm91 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danieldietzler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rithask commented May 24, 2024

mertalev commented May 24, 2024

ThomasConrad commented Jun 7, 2024

alextran1502 commented Jun 7, 2024

Amerlander commented Jun 22, 2024 • edited Loading

NicholasFlamy commented Jun 22, 2024

alextran1502 commented May 16, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented May 16, 2024 •

edited

Loading

NicholasFlamy commented May 17, 2024 •

edited

Loading

NicholasFlamy commented May 18, 2024 •

edited

Loading

Amerlander commented Jun 22, 2024 •

edited

Loading