Faceset logic?

[vegas70]

How does the faceset work? Do I have to put faces in different positions? If I put more versions of a frontal face is there an improvement or is it useless?

thanks

[dastommygun]

I'm also interested in this...

[C0untFloyd]

#656

[C0untFloyd]

There is no magical algorithm behind, it's just an average of all the face embeddings:

roop-unleashed/roop/FaceSet.py

Line 14 in f0950c2

def AverageEmbeddings(self):

This of course means, if you use too many images, some special face features will be "smoothed out" by this averaging.

[ItsMeTho]

Very enlightening.

I've wondered how to tackle let's say a turning face.. at some point beyond side view it's.. gone. Any ideas?

[Elise96nl]

you know what is strange. i have 100+ facesets right now. Most of them are okish. there is one that works so perfectly. but the source material for that one is mostly low res. maybe its just an easy face. but im amazed every time. every angle, always looks good. wish all of them where that good.

will look into it tommorow. and give you guys a summary, why that one is so good. but it was 8/10 faces inside the set.

[Elise96nl]

my perfect model is 3.1 mb

consists of 14 pictures, but 11 unique ones. 3 are double.

but and here is the thing that makes it so good i quess. 6 pictures are almost identical, but like 2 or 3 degrees different. i made them of that person myself, and made like 20 in a few sec, while moving my phone a little bit. So i quess that's the big secret. Not 10 random high res pics, but same moment, same camera, same lightning and slightly different angles

[ItsMeTho]

Wow.. that's pretty clever..

I know this technique. I use it to steer outcome of cloners in Cinema 4D. Especially useful for vegetation. It never occurred to me to use it here. Thanks!

[Elise96nl]

also the more brutal angles are really great with only this faceset. It works 99 %.

I have no idea how AI/Smart this insight stuff actually is. But this model performing so wel. It's almost like they have some kind of extrapolate smart shit working.

Like my face set is frontview and + 5 degrees and + 15 degrees to right and left. Its like that's exactly the data it needs to extrapolate almost every angle.

it also has something to do with the landmark codes.

it needs them all to register on 1 of the pics in the set.

https://github.com/nttstar/insightface-resources/blob/master/alignment/images/2d106markup.jpg

(edited your URL to work)

[ItsMeTho]

From tests I've found that the more typical the face is, the easier it gets. Pretty faces just seem to do well. With a unique face I found it works better not to have so many different photos. Just my observation.

There could be tradeoffs worth considering if you really going for it and you have trouble..
If u have access to really good noise reduction (NR) like Davinci Resolve has (paid), you can allow more faces, Roop make more mistakes.. and smooth it all out later, at expense of sharpness, depending on how far you take NR.

But it can help. I tested with just 1 image (frontal) and it worked nicely, but u get the same face all the time, if u know what I mean.

With many variations I could catch much of the expressions. I think if all from the same moment works best but not sure of anything haha. Anyway, tradeoff was a lot of (subtle) flickering in the face (jumpy at times) that I completely tighened up and smoothed out in Davinci Resolve. I got a super steady face back that looked much more 'alive'.

But u have to post process that again to not end up with a too much of a clean image. At very low settings DR already takes care all regular noise, just a bit higher your grain is gone, and higher settings take care of flickering and such.

Reminds me of autotune. Low settings you fix your false notes, in overdrive u get Cher :)

Unfortunately it doesn't detect BRUTAL frames yet to take those out. That would be so cool. I have a controller with a wheel for editing in DR timeline so going through Roop video to take out those frames is blazing fast compared mouse+keyboard. But it's a chore nonetheless if u have a longer video.