Import face regions from propietary metadata formats

[bugfest]

@whenever OP: #6455 (comment)

[bugfest]

@whenever I'm preparing a script in NodeJS so that we can use the same library (exitfool-vendored) Immich is using to parse metadata. I'll post it here as soon as I have a first version ready

[bugfest]

Issue (photostructure/exiftool-vendored.js#211) reported and PR/fix (photostructure/exiftool-vendored.js#212) requested

[whenever]

Hi @bugfest: just going over the questions you answered a few days ago

One name per rectangle

_> 4. Probably have to add RegionAreaUnit tag

normalized coordinates are the expected for now_

By this I assume you mean that the program that reads the Region metadata into Immich assumes Normalized and therefore the tag does not need to be there?

> Then import your pictures (unchanged) with these new XMP metadata sidecars so that you can keep both in sync easily

So we would leave the metadata that is embedded jpg files alone and create separate sidecar files with the equivalent MWG regions metadata. It would be easy to see which has changed and either update the ACDsee embedded or the MWG Sidecar? Should we do the same for the other metadata such as categories? Does Immich look at both embedded and sidecar and if so how does it decide which to use. Maybe we should consider duplicating all the metadata in the sidecar with the few changes needed for MWG regions.

On the NodeJS, as I said I am not a coder. I know what NodeJS is conceptually so I may need a little help with that.

[bugfest]

_> 4. Probably have to add RegionAreaUnit tag

Yes, specifically RegionInfo.RegionList[].Area.Unit

normalized coordinates are the expected for now_

By this I assume you mean that the program that reads the Region metadata into Immich assumes Normalized and therefore the tag does not need to be there?

If already normalized, is an optional field

> Then import your pictures (unchanged) with these new XMP metadata sidecars so that you can keep both in sync easily

So we would leave the metadata that is embedded jpg files alone and create separate sidecar files with the equivalent MWG regions metadata. It would be easy to see which has changed and either update the ACDsee embedded or the MWG Sidecar?

I'd say the safest is the sidecar file, since can be recreated from the original image and you wouldn't loose your existing face/regions if some error happens during the transformation

Should we do the same for the other metadata such as categories? Does Immich look at both embedded and sidecar and if so how does it decide which to use. Maybe we should consider duplicating all the metadata in the sidecar with the few changes needed for MWG regions.

We can look into it when we solve the regions issue

[bugfest]

Hi @whenever!

I've managed to get the script working. I've published it at https://github.com/bugfest/exif-companion

I had some issues with exif-vendored library due to an internal issue when serializing the face regions into the metadata, I believe it's a bug of this library and I'll open an Issue/PR on their repo. In the mean time I've forked the library and published a branch with the fix we need.

Here you have an screenshot of your test ACDSee picture after import it with the sidecar generated by my script:

I'm pretty sure the script can be improved to automate the generation of the modified XMP files. I have some ideas but before implementing them I'd like you to verify this approach works for you.

[bo0tzz]

Nice work! Seems like you've gone for a general-purpose project rather than something Immich specific (which I like to see!), but feel free to PR adding a link to the https://immich.app/docs/community-projects page if you'd like.

[bugfest]

Thanks for the suggestion @bo0tzz! as soon as we prove it works and refine it a bit I'll request it to be added

[whenever]

Hi @bugfest: What you show in the attached PNG looks great. The approach sounds great, in fact it looks like you've done what I'm looking for.

I'm not sure how I set it up on my system to run.

I've installed NodeJS, cloned exif-companion and then ran it using my test picture from acdsee. Had a few issues involving module yargs not found, but I think I resolved those.

When I run it I get an .xmp file but as best as I can see, on my end it is still using ACDSee regions. In my case the face regions do not show up in DigiKam or in Immich when I import the processed picture.

I think I'm missing something. Do you want me to run something on my system?

[bugfest]

Hi @whenever, thanks for trying it out!

Can you run the command with the --debug flag enabled and share the output? you can try with the files you shared with me. Let me show you the steps I followed for reference:

My setup:

• I have your test files in <HOME>/Pictures/test test folder. I created a new one with the fixed .xmp files at <HOME>/Pictures/test-fix
• In Immich, I've added <HOME>/Pictures/test-fix as external library

Process:

1. Convert the ACDSee XMP file to an Immich-compatible one with the following commands. I enclose the resulting folder for reference (test-fix.zip):

cd exif-companion
npm i
npm run start -- --input "<HOME>/Pictures/test/Friends Test pictureAcdsee.jpg.xmp" --output "<HOME>/Pictures/test-fix/Friends Test pictureAcdsee.jpg.xmp" --keep --debug

2. Copy the original jpg image to the test-fix folder:

cp "<HOME>/Pictures/test/Friends Test pictureAcdsee.jpg" "<HOME>/Pictures/test-fix/Friends Test pictureAcdsee.jpg"

3. In Immich:
   3.1 Make sure the image is not already in your library. Also, empty the trash just in case.
   3.2 Go to the gallery and run a re-scan
   3.3. The new picture should appear with the face regions properly tagged.

[whenever]

Hi @bugfest:

I've included a Zip of inputs, outputs and a txt of the Powershell run with debug enabled.
ForBugfest.zip

Comparing the 2 output.xmp files, the one you sent and the one in this zip, they start out similar but after a page it becomes unreadable.

My set up is as follows:

Windows 11 machine

• NodeJS installed, exif-companion in windows folder, run from powershell and picture and xmp in the same windows folder
• Output.xmp has a different name from the input jpg, not sure how immich would know they go together
• Docker running Immich: Using the import feature (I've not yet set up external libraries), no other pictures and emptied trash.

Cheers:

Whenever

[bugfest]

Hi @whenever, thanks for the files. From your CLI output I see you're using an JPG as input and the output is expected to XMP. In this initial version I'm expecting both input and output to be XMP.

Looking into your process to import the image into Immich, it's expected the XMP data is not imported. AFAIK when uploading the image to Immich directly, the server won't be able to parse the XMP file; that's why I was using an external library; when Immich server scans the external library folder it will find both JPG and XMP sidecar (named as: filename.jpg and filename.jpg.xmp).

I'm going to look into modifying exif-companion so that can support image files as input & output. With this you should be able to import a modified JPG directly into Immich so that it imports the face regions (without XMP sidecar). The downside of this case is that you'll have to store two versions of the image (ACDSee original and modified), unless you want to discard the modified one just after upload it to Immich.

In the mean time, if you want to try out my process, try modifying the XMP file that ACDSee generates with my tool and setup the external library to import both the original JPG and the modified XMP.

[bugfest]

Just found out exif-companion just works for JPGs as well. I spotted an issue with the orientation of the image though, digging into it now.

[whenever]

Hi @bugfest

I've been planning to set up external libraries on Immich as it is how I want to deal with things in the longterm. I've just not got round to figuring that out. I will do that now. The XMP input file, are you generating that with exiftool and if so are there any special parameters. Normally I just drag the files onto specific exe files (exiftool(-a -u -g1 -w xmp).exe).

I'll report back once I've tried it using external libraries

[bugfest]

Hi @whenever, just tested you can get it working with your process, you just need to use a .jpg output filename. I've also just published a new version of the tool if you want to update your local install.

I'm working on another update for exif-companion tool to use upstream's exiftool-vendored.js since my fix/PR got merged (photostructure/exiftool-vendored.js#212) some minutes ago.

[whenever]

Hi @bugfest:

I've got the external libraries set up on Immich. The facial import works well on the DigiKam jpg's. I've updated exif-companion. I've tried this 3 ways

1. Extract sidecar XMP file from jpg using "exiftool(-a -u -g1 -w xmp).exe" and using the xmp file as input . In this case the xmp output is exactly the same as the xmp input.
2. Extract sidecar XMP file from jpg using "exiftool(-k -o %d%f.xmp).exe" and using the xmp file as input . In this case the xmp output is different from the xmp input.
3. Just using .jpg as input and xmp file is created

None of these case are recognized by Immich and they all show unrecognized faces. I seem to remember that jpg which have facial metadata would not be processed by Immich facial recognition.

[whenever]

Hi @bugfest

This is fantastic

I reread your last message, I tried it by just using the .jpg for the input (.i.e. no sidecar at all) and it imported the names and regions, thumbnail is not working but I think that's a similar issue that I had before.

I must be doing something wrong when I extract the XMP file using exiftool.

[bugfest]

Hi @whenever, I think it worked. The issue you're describing is not due to a problem of the image but the thumbnails generation. Go to Immich Admin panel, jobs and run thumbnails generation. You should be all set!

[whenever]

Hi @bugfest:

I am quite excited, the .jpg input/output filename works well and refreshing the thumbnails fixed the issue.

I would prefer to use the XML sidecar method that you are using as it does keep my original files unchanged, however I am at a loss to figure out how to get it working. I've tried variations of exiftool directly, variations of DigiKam writing to xmp files using exiftool and not using exiftool. No success. How did you create the xmp sidecar that is then processed by exic-companion? Your plans on how you will deploy the final product may make this issue moot.

As to process:

1. What are your thoughts on exif-tool companion? Would it be deployed on its own or as part of Immich?
2. I would be happy with just being able to drop a folder into an exec and it would process all the pictures leaving the folder structure intact and placing xmp files in the same folder with the new name.
3. My preferred process would be exif-tool reads embedded metadata from the jpg and writes out all the Metadata to a new XMP file with the appropriate adjustments to face regions. The pre-existence of an XMP file could then be used to trigger a sync if you want.
4. My current plans are to keep my current ACDSee Picture Archive separate and do a oneway synch to the Immich version and then process with exif-companion
5. If exif-companion is deployed as part of Immich (even better from my view), then ACDSee Jpegs could trigger the exif-companion during the metadata import (as far as I can tell ACDSee does not have a sidecar export capability at this time, so an existing xmp file could be used to trigger a sync.

Thanks for all you great work so far

Whenever

[bugfest]

Hi @whenever,

I am quite excited, the .jpg input/output filename works well and refreshing the thumbnails fixed the issue.

Nice!! Well done!

I would prefer to use the XML sidecar method that you are using as it does keep my original files unchanged, however I am at a loss to figure out how to get it working. I've tried variations of exiftool directly, variations of DigiKam writing to xmp files using exiftool and not using exiftool. No success. How did you create the xmp sidecar that is then processed by exic-companion? Your plans on how you will deploy the final product may make this issue moot.

As to process:

1. What are your thoughts on exif-tool companion?  Would it be deployed on its own or as part of Immich?

I don't think exif-companion will be part of Immich deployment. In the last update I defined a Dockerfile to let users use Docker instead of the nodejs/npm install process. If you're already hosting Immich using docker this might be as good.

2. I would be happy with just being able to drop a folder into an exec and it would process all the pictures leaving the folder structure intact and placing xmp files in the same folder with the new name.

Ok, I can explore allowing exif-companion to process all the files of a folder.

3. My preferred process would be exif-tool reads embedded metadata from the jpg and writes out all the Metadata to a new XMP file with the appropriate adjustments to face regions.  The pre-existence of an XMP file could then be used to trigger a sync if you want.

I suggest you create a script that fires exif-companion to process the folder(s) images and then triggers the library rescan via Immich's API (or you trigger it manually). I don't have plans to extend exif-companion to support Immich API. I'd like this tool to be as generic as possible otherwhise will be very hard to maintain.

4. My current plans are to keep my current ACDSee Picture Archive separate  and do a oneway synch to the Immich version and then process with exif-companion

Understood. It's a good process.

5. If exif-companion is deployed as part of Immich (even better from my view), then ACDSee Jpegs could trigger the exif-companion during the metadata import (as far as I can tell ACDSee does not have a sidecar export capability at this time, so an existing xmp file could be used to trigger a sync.

I would trigger the sync after exif-companion finishes as suggested above.

Thanks for all you great work so far

Whenever

Happy to help : )

I'll start working on the folder/glob processing feature as soon as I have some free time. I'll keep you posted

[whenever]

Hi @bugfest

I've gotten a bat file written that goes through all the .jpg files in a directory and process them through exif-companion. I did all my testing using the test pictures that I sent you, When I got to the point of using my actual pictures, exif-companion gives errors and does not write out the mwg regions. In the same bat run, the test files always work.

I've attached 2 txt of a short run of jpg, one with and one without --debug
RunResults.zip

I can send you the bat file if you like, its very simple

[bugfest]

Thanks @whenever Sorry I've been out for a while. I have the recursive implementation half-backed. I'll focus on these errors first though. Thanks for the logs!

[whenever]

Hi @bugfest

There are at least 2 types of errors, I've managed to track down one of them (this one accounts for most of the errors I've seen), and at least one cause is that there is a categories: line in the embedded xmp data (could be other causes too).

I made the following changes to xmp.ts on my own branch around 74

const partialMetadata = {
RegionInfo: metadata.RegionInfo
};

await exiftool.write(outputFilePath, partialMetadata)

While this may effect the --keep option exif-companion, files that have categories are processed and the facial data shows up on Immich

Still trying to track down the cause of the other error type.

Cheers

Whenever

[whenever]

Hi @bugfest

I've tracked down the cause of the second error type. Occasionally ACDSee does not include the ALGAreas only the DLY Areas. I changed XMP.ts to use DLY instead and they all worked. I will run a batch with a lot more pictures to see how it goes.

[bugfest]

Hi @whenever would you say the DLY provide better face regions than ALGAreas? If not I can implement a preference on ALGAreas and failback to DLY if not present

[whenever]

From my perspective either one will work. The region rectangle on Immich is only important in 2 situations: 1) for the thumbnail of the person (you can usually find on photo that works) and 2) to identify the person when you hover over the thumbnail. All the ones I looked at with DLY were fine.

I have no idea why ACDSee does not always output both ALG and DLY. Some of the regions did not have ALG in the small batch of files I tested at the beginning. Using DLY on my branch, I had no failures using the 500+ files I ran yesterday and all files that failed where of the 2 types I discussed earlier Missing name or no regions at all, in the first case I can fix that within ACDSee and in the second it has no real impact as there are no faces anyways.

[whenever]

Hi @bugfest

I know that this is not high priority. What you've done with exif-companion is gold for me and I can't thank you enough.

I am going to continue looking for, and likely finding, error conditions for exif-companion and, with a lot of help for copilot, fix them on my branch if I can or develop a workaround.

My question to you, do you want to hear about the issues I find with exif-companion? or should I continue on in silence?

Is there a better way to let you know about these issues? I'm not familiar with github to even be sure that you are still getting these notes.

My latest discovery
I did a run of some 500+ ACDSee files (my 2 fixes above implemented) and got some 9 .jpg files failing. Here are the causes that I've found for the current batch of failures

1. No name for a face region, fails (2 of 9). ACDsee writes out the region info but there is no "Name" or NameAssignType" fields. For me I will fix this issue in my ACDSee library as they occur. Current thoughts are to just make them Unknown. I may attempt a proper by adding a condition check in XMP.ts for this condition.
2. No faces in the picture (7 of 9). ACDSee writes out the "RegionInfoACDSee" but then no "RegionList". For me I will likely just add a ghost face region. I may attempt a proper by adding a condition check in XMP.ts for this condition.

I will keep adding discoveries here unless you tell me otherwise.

Cheers

Whenever

[bugfest]

Hi @whenever, this feedback is very valuable. I'll create proper error handling for those cases