Dataset

The CelebAMask-HQ dataset was used to train the skin segmentation model Steps to prepare the dataset:

1. Download the .zip file from the CelebAMask-HQ repo (or use this link directly: link
2. Place the CelebAMask-HQ.zip file in the projects root directory
3. Run data preparation script: python src/bin/prepare_data.py

Skin segmentation model

The PSPNet model was used for skin segmentation task (trained from scratch on CeleAMask-HQ dataset). Run the following line to train the model (assuming that dataset is already prepared) python src/bin/celebA/train.py

rPPG signal extraction

To extract rPPG signal from the face:

1. Segment all video frames to find the skin pixels for each frame
2. Apply erosion to make the skin mask a bit smaller to prevent mask leak out of the face
3. Average skin pixels for R, G and B channels

Pulse estimation

1. Extract rPPG from video - after this step the R, G and B signals are present (signal length is equal to number of frames)
2. Filter out pixels belonging to the skin
3. Apply Plane Orthogonal to the Skin (POS) algorithm to get POS signal from the RGB signals
4. Run FFT for RGB channels of rPPG signals to find the strongest frequency (which corresponds to the Heart Rate)
5. Calculate the Heart Rate using the strongest frequency: HR = strongest_freq * 60

Results

Processing pipeline

1. Raw frame
2. Frame trasformed to match model input size
3. Model output segmentation map (with all classes)
4. Output segmentation map filtered to match skin and nose labels (skin_mask)
5. Raw frame with pulse extracted for each pixel and masked with skin_mask

pulse_extraction_from_skin_hand_good_light.mp4

RGB and POS signals along with the frequency spectrum and estimated Heart Rate (HR).