[Feature] Person identification and manual tagging

[Atemu]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Immich currently implements automatic face detection which is great. However, not all faces can reasonably be detected or even are detectable as the face may simply not be visible in many cases.

As a user however, I do not actually care whether a face is detected or visible when filtering for a specific person; what I actually care about is whether that specific person is in the picture or not. For this purpose, it'd be great to have an additional mechanism for detecting persons. This mechanism would of course take face detection as an input as a face is quite the identifying factor but also other body parts or perhaps even clothing and other wearable objects.

Independently of automatic detection, the user should be able to tag people manually as no detection mechanism will ever be perfect. Imperfect automatic detection should not prevent sensible data filtering when manual classification is a possibility. If a person I care about is not detected in some of the pictures of them, you should be able to manually tag them and have them show up when I search for their name.
Ideally, the detection mechanism would learn from these manual classifications similar to paperless' excellent document classification system.

MVP: Departure from person == face to person == face or manual tag
Automatic solution: Person identification based on a soft algorithm (i.e. machine learning or genetic algorithm) + manual tagging
Ideal solution: Automatic solution + automatic learning a la Paperless

I am not currently aware of a ready-to-use mechanism that can achieve what I proposed.

Doing some research online, this feels like it should be in the realm of possibility. For instance, I found https://docs.plumerai.com/1.16/face_identification_overview/ who claim to be able to do real time person localisation + face detection on a single core ARM Cortex-A72. It's not quite what we're looking for here but it's not too far off.
Does anyone know of something perhaps?

Platform

• Server
• Web
• Mobile

[bo0tzz]

#7301

[Atemu]

That is similar but it'd still entirely rely on faces for people detection. I'd like for Immich to get away from the notion that every person must have their face visible a picture to be prominently visible in said picture because that's simply not the case. It's easy to imagine pictures where the primary subject has no visible face.

If I filter Immich by a person, I'd like for it to show all pictures where that person is present. Even if it's just a close up of them holding something in their hand, I would like it to be directly associated with that person; them being "in" the picture. That specific example will likely require a manual tag for the foreseeable future but less extreme cases could perhaps be detected automatically.

[Atemu]

One thing I didn't realise when writing the OP is that, while the model I found cannot identify a specific person without a face, it can at least identify that there is some person in the picture and, more importantly, where.

This could be very useful for aiding manual detection as you could combine the bounding box of a person with the existing face detection to detect cases where a person was detected but no face: If the detected face and the bounding box do share a significant amount of area, the detected person must be the one identified by the face but if no face is detected within a person's bounding box, that would signify a picture in which the user would need to tag the person manually.

This system could also aid the prevention of false-positives in face detection as a face outside of a person's bounding box should be unlikely; potentially allowing for a lower minimum detection score.

[radh21301]

I just posted a comment in (#2472) about this. And realized this one is the exact feature I described there.

I certainly agree! It would be so awesome for Immich to do something like this.