Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow non-read-only external libraries that support asset deletion #8943

Closed
wants to merge 3 commits into from
Closed

Allow non-read-only external libraries that support asset deletion #8943

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
11cade0
feat(server): add isReadOnly property to the Library model
valpackett Apr 8, 2024
c9f82db
feat(server,web): allow deletion in external libraries marked not-rea…
valpackett Apr 20, 2024
49ac546
chore(server): add test for deleting external asset
valpackett Apr 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions mobile/openapi/doc/UpdateLibraryDto.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 18 additions & 1 deletion mobile/openapi/lib/model/update_library_dto.dart
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions mobile/openapi/test/update_library_dto_test.dart
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions open-api/immich-openapi-specs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8778,6 +8778,10 @@
},
"type": "array"
},
"isReadOnly": {
"nullable": true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this shouldn't be nullable either

"type": "boolean"
},
"name": {
"type": "string"
},
Expand All @@ -8803,6 +8807,7 @@
"exclusionPatterns",
"id",
"importPaths",
"isReadOnly",
"name",
"ownerId",
"refreshedAt",
Expand Down Expand Up @@ -11183,6 +11188,9 @@
},
"type": "array"
},
"isReadOnly": {
"type": "boolean"
},
"isVisible": {
"type": "boolean"
},
Expand Down
1 change: 1 addition & 0 deletions open-api/typescript-sdk/src/fetch-client.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,7 @@ export type CreateLibraryDto = {
export type UpdateLibraryDto = {
exclusionPatterns?: string[];
importPaths?: string[];
isReadOnly?: boolean;
isVisible?: boolean;
name?: string;
};
Expand Down
6 changes: 6 additions & 0 deletions server/src/dtos/library.dto.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ export class UpdateLibraryDto {
@ValidateBoolean({ optional: true })
isVisible?: boolean;

@ValidateBoolean({ optional: true })
isReadOnly?: boolean;

@Optional()
@IsString({ each: true })
@IsNotEmpty({ each: true })
Expand Down Expand Up @@ -128,6 +131,8 @@ export class LibraryResponseDto {
createdAt!: Date;
updatedAt!: Date;
refreshedAt!: Date | null;

isReadOnly!: boolean | null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't nullable.

}

export class LibraryStatsResponseDto {
Expand Down Expand Up @@ -160,5 +165,6 @@ export function mapLibrary(entity: LibraryEntity): LibraryResponseDto {
assetCount,
importPaths: entity.importPaths,
exclusionPatterns: entity.exclusionPatterns,
isReadOnly: entity.isReadOnly,
};
}
3 changes: 3 additions & 0 deletions server/src/entities/library.entity.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ export class LibraryEntity {

@Column({ type: 'boolean', default: true })
isVisible!: boolean;

@Column({ type: 'boolean', nullable: true })
isReadOnly!: boolean | null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be nullable and should default to true

}

export enum LibraryType {
Expand Down
14 changes: 14 additions & 0 deletions server/src/migrations/1712293452361-AddLibraryReadOnly.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { MigrationInterface, QueryRunner } from "typeorm";

export class AddLibraryReadOnly1712293452361 implements MigrationInterface {
name = 'AddLibraryReadOnly1712293452361'

public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query(`ALTER TABLE "libraries" ADD "isReadOnly" boolean`);
}

public async down(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query(`ALTER TABLE "libraries" DROP COLUMN "isReadOnly"`);
}

}
8 changes: 7 additions & 1 deletion server/src/queries/library.repository.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ FROM
"LibraryEntity"."deletedAt" AS "LibraryEntity_deletedAt",
"LibraryEntity"."refreshedAt" AS "LibraryEntity_refreshedAt",
"LibraryEntity"."isVisible" AS "LibraryEntity_isVisible",
"LibraryEntity"."isReadOnly" AS "LibraryEntity_isReadOnly",
"LibraryEntity__LibraryEntity_owner"."id" AS "LibraryEntity__LibraryEntity_owner_id",
"LibraryEntity__LibraryEntity_owner"."name" AS "LibraryEntity__LibraryEntity_owner_name",
"LibraryEntity__LibraryEntity_owner"."avatarColor" AS "LibraryEntity__LibraryEntity_owner_avatarColor",
Expand Down Expand Up @@ -90,7 +91,8 @@ SELECT
"LibraryEntity"."updatedAt" AS "LibraryEntity_updatedAt",
"LibraryEntity"."deletedAt" AS "LibraryEntity_deletedAt",
"LibraryEntity"."refreshedAt" AS "LibraryEntity_refreshedAt",
"LibraryEntity"."isVisible" AS "LibraryEntity_isVisible"
"LibraryEntity"."isVisible" AS "LibraryEntity_isVisible",
"LibraryEntity"."isReadOnly" AS "LibraryEntity_isReadOnly"
FROM
"libraries" "LibraryEntity"
WHERE
Expand Down Expand Up @@ -133,6 +135,7 @@ SELECT
"LibraryEntity"."deletedAt" AS "LibraryEntity_deletedAt",
"LibraryEntity"."refreshedAt" AS "LibraryEntity_refreshedAt",
"LibraryEntity"."isVisible" AS "LibraryEntity_isVisible",
"LibraryEntity"."isReadOnly" AS "LibraryEntity_isReadOnly",
"LibraryEntity__LibraryEntity_owner"."id" AS "LibraryEntity__LibraryEntity_owner_id",
"LibraryEntity__LibraryEntity_owner"."name" AS "LibraryEntity__LibraryEntity_owner_name",
"LibraryEntity__LibraryEntity_owner"."avatarColor" AS "LibraryEntity__LibraryEntity_owner_avatarColor",
Expand Down Expand Up @@ -179,6 +182,7 @@ SELECT
"LibraryEntity"."deletedAt" AS "LibraryEntity_deletedAt",
"LibraryEntity"."refreshedAt" AS "LibraryEntity_refreshedAt",
"LibraryEntity"."isVisible" AS "LibraryEntity_isVisible",
"LibraryEntity"."isReadOnly" AS "LibraryEntity_isReadOnly",
"LibraryEntity__LibraryEntity_owner"."id" AS "LibraryEntity__LibraryEntity_owner_id",
"LibraryEntity__LibraryEntity_owner"."name" AS "LibraryEntity__LibraryEntity_owner_name",
"LibraryEntity__LibraryEntity_owner"."avatarColor" AS "LibraryEntity__LibraryEntity_owner_avatarColor",
Expand Down Expand Up @@ -219,6 +223,7 @@ SELECT
"LibraryEntity"."deletedAt" AS "LibraryEntity_deletedAt",
"LibraryEntity"."refreshedAt" AS "LibraryEntity_refreshedAt",
"LibraryEntity"."isVisible" AS "LibraryEntity_isVisible",
"LibraryEntity"."isReadOnly" AS "LibraryEntity_isReadOnly",
"LibraryEntity__LibraryEntity_owner"."id" AS "LibraryEntity__LibraryEntity_owner_id",
"LibraryEntity__LibraryEntity_owner"."name" AS "LibraryEntity__LibraryEntity_owner_name",
"LibraryEntity__LibraryEntity_owner"."avatarColor" AS "LibraryEntity__LibraryEntity_owner_avatarColor",
Expand Down Expand Up @@ -259,6 +264,7 @@ SELECT
"libraries"."deletedAt" AS "libraries_deletedAt",
"libraries"."refreshedAt" AS "libraries_refreshedAt",
"libraries"."isVisible" AS "libraries_isVisible",
"libraries"."isReadOnly" AS "libraries_isReadOnly",
COUNT("assets"."id") FILTER (
WHERE
"assets"."type" = 'IMAGE'
Expand Down
2 changes: 1 addition & 1 deletion server/src/repositories/asset.repository.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ export class AssetRepository implements IAssetRepository {

@Chunked()
async softDeleteAll(ids: string[]): Promise<void> {
await this.repository.softDelete({ id: In(ids), isExternal: false });
await this.repository.softDelete({ id: In(ids) });
}

@Chunked()
Expand Down
33 changes: 33 additions & 0 deletions server/src/services/asset.service.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -663,7 +663,7 @@

await sut.handleAssetDeletion({ id: assetStub.readOnly.id });

expect(jobMock.queue.mock.calls).toEqual([

Check failure on line 666 in server/src/services/asset.service.spec.ts

View workflow job for this annotation

GitHub Actions / Server

src/services/asset.service.spec.ts > AssetService > handleAssetDeletion > should only delete generated files for readonly assets 

AssertionError: expected [ [ { name: 'delete-files', …(1) } ] ] to deeply equal [ [ { name: 'delete-files', …(1) } ] ]

- Expected
+ Received

  Array [
    Array [
      Object {
        "data": Object {
          "files": Array [
            null,
            "/uploads/user-id/thumbs/path.ext",
            null,
+           "/original/path.ext.xmp",
+           "/original/path.ext",
          ],
        },
        "name": "delete-files",
      },
    ],
  ]

 ❯ src/services/asset.service.spec.ts:666:40
[
{
name: JobName.DELETE_FILES,
Expand Down Expand Up @@ -691,6 +691,39 @@
expect(assetMock.remove).not.toHaveBeenCalled();
});

it('should process assets from external non-read-only library without fromExternal flag', async () => {
when(assetMock.getById)

Check failure on line 695 in server/src/services/asset.service.spec.ts

View workflow job for this annotation

GitHub Actions / Server

src/services/asset.service.spec.ts > AssetService > handleAssetDeletion > should process assets from external non-read-only library without fromExternal flag 

ReferenceError: when is not defined
 ❯ src/services/asset.service.spec.ts:695:7
.calledWith(assetStub.externalDeletable.id, {
faces: {
person: true,
},
library: true,
stack: { assets: true },
exifInfo: true,
})
.mockResolvedValue(assetStub.externalDeletable);

await sut.handleAssetDeletion({ id: assetStub.externalDeletable.id });

expect(assetMock.remove).toHaveBeenCalledWith(assetStub.externalDeletable);
expect(jobMock.queue.mock.calls).toEqual([
[
{
name: JobName.DELETE_FILES,
data: {
files: [
assetStub.externalDeletable.thumbnailPath,
assetStub.externalDeletable.previewPath,
assetStub.externalDeletable.encodedVideoPath,
assetStub.externalDeletable.sidecarPath,
assetStub.externalDeletable.originalPath,
],
},
},
],
]);
});

it('should process assets from external library with fromExternal flag', async () => {
assetMock.getById.mockResolvedValue(assetStub.external);

Expand Down
8 changes: 5 additions & 3 deletions server/src/services/asset.service.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,8 +372,10 @@ export class AssetService {
return JobStatus.FAILED;
}

// Ignore requests that are not from external library job but is for an external asset
if (!fromExternal && (!asset.library || asset.library.type === LibraryType.EXTERNAL)) {
// Ignore requests that are not from external library job but is for an external read-only asset
const isReadOnlyLibrary = !asset.library || asset.library.isReadOnly ||
(asset.library.isReadOnly === null && asset.library.type === LibraryType.EXTERNAL);
if (!fromExternal && isReadOnlyLibrary) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should just check if from external and the asset is read only and skip if so. No reason to make it complicated.

return JobStatus.SKIPPED;
}

Expand Down Expand Up @@ -406,7 +408,7 @@ export class AssetService {
}

const files = [asset.thumbnailPath, asset.previewPath, asset.encodedVideoPath];
if (!(asset.isExternal || asset.isReadOnly)) {
if (!isReadOnlyLibrary) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If asset isn't readonly

files.push(asset.sidecarPath, asset.originalPath);
}

Expand Down
40 changes: 40 additions & 0 deletions server/test/fixtures/asset.stub.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,46 @@ export const assetStub = {
} as ExifEntity,
}),

externalDeletable: Object.freeze<AssetEntity>({
id: 'asset-id',
deviceAssetId: 'device-asset-id',
fileModifiedAt: new Date('2023-02-23T05:06:29.716Z'),
fileCreatedAt: new Date('2023-02-23T05:06:29.716Z'),
owner: userStub.user1,
ownerId: 'user-id',
deviceId: 'device-id',
originalPath: '/ext/photo.jpg',
previewPath: '/uploads/user-id/thumbs/path.jpg',
checksum: Buffer.from('file hash', 'utf8'),
type: AssetType.IMAGE,
thumbnailPath: '/uploads/user-id/webp/path.ext',
thumbhash: Buffer.from('blablabla', 'base64'),
encodedVideoPath: null,
createdAt: new Date('2023-02-23T05:06:29.716Z'),
updatedAt: new Date('2023-02-23T05:06:29.716Z'),
localDateTime: new Date('2023-02-23T05:06:29.716Z'),
isFavorite: true,
isArchived: false,
isReadOnly: false,
isExternal: true,
duration: null,
isVisible: true,
livePhotoVideo: null,
livePhotoVideoId: null,
isOffline: false,
libraryId: 'library-id',
library: libraryStub.externalLibraryNotReadOnly,
tags: [],
sharedLinks: [],
originalFileName: 'asset-id.jpg',
faces: [],
deletedAt: null,
sidecarPath: null,
exifInfo: {
fileSizeInByte: 5000,
} as ExifEntity,
}),

offline: Object.freeze<AssetEntity>({
id: 'asset-id',
deviceAssetId: 'device-asset-id',
Expand Down
Loading
Loading