This is the python implementation of this paper
This is abstract directly quated from paper
We extract heart rate and beat lengths from videos by measuring subtle head motion caused by the Newtonian reaction to the influx of blood at each beat. Our method tracks features on the head and performs principal compo- nent analysis (PCA) to decompose their trajectories into a set of component motions. It then chooses the component that best corresponds to heartbeats based on its temporal frequency spectrum. Finally, we analyze the motion pro- jected to this component and identify peaks of the trajecto- ries, which correspond to heartbeats. When evaluated on 18 subjects, our approach reported heart rates nearly iden- tical to an electrocardiogram device. Additionally we were able to capture clinically relevant information about heart rate variability.
Base of this implementation is made in 3 days for a hackathon, so there are still plenty of bugs and ugly hacks.
Sampling Frequency or FPS for video is important for getting accurate results from FFT and Bandpass filter.
All push requests and questions are welcome.
For using algorithm from paper:
python main.py --graph same as
python main.py -c 0 --face_detector haar --graph If you want use improved version with face landmarks:
python main.py --face_detector face_shape
If you want to use video file instead:
python main.py -f path
If you want to test modules:
python modules/face.py
python modules/tracking.py
Project consists of 3 modules for now:
- Face - FacePoints
- tracking - TrackPoints
- signal_processing - SignalProcess
├── main.py
├── modules
│ ├── face.py
│ ├── signal_processing.py
│ └── tracking.py
├── papers
│ ├── Balakrishnan_Detecting_Pulse_from_2013_CVPR_paper.pdf
│ ├── electronics-08-00663.pdf
├── README.md
├── requirements.txt
├── tests
│ ├── common.py
│ ├── dlib_dedect.py
│ ├── fft.ipynb
│ ├── lk_track.py
│ ├── signal_pross.ipynb
│ └── vid_test.py
├── data
│ ├── haarcascade
│ │ ├── ***
│ └── shape_predictor_68_face_landmarks.dat
This module is for detecting face and getting trackable points on face. You can select detector with dedector_type argument.
- haar
- OpenCV's Haar face detection, gets face rectangle and rescales, then removes eyes from area
- Finds optically trackable points in that filtered area
- Fastest one
- dlib
- Same as Haar, but this time uses Dlib's face detection for face rectangle
- More accurate
- face_shape
- Finds 68 face landmarks on face. Then removes eyes and mount landmark points for getting more stabilized data
- Slow but more accurate
This module is responsible for tracking points throughout video.
Takes
face_detector, max_trace_num=150, max_trace_history=60 as input.
If Haar or Dlib is used on face detection then tracker checks every point with backtracking. If a point is not consistent with forward and backtracking, then trace is marked as faulty and and gets deleted.
On every update non-faulty traces are counted.
If their number is not equal to max_trace_num then new trackable points are requested from face_detector.
Higher the trace number, higher the CPU power needed.
If face_shape is used on face detection, then initial landmark points are tracked again with opticalflow.
If a point on face landmarks marked as faulty then face landmarks are requested from face_detector but only faulty points are replaced.
Because face landmarks only gives out integer coordinates, not using opticalflow will result losing details. Opticalflow tracks points on subpixel level and returns float coordinates.
This module is responsible for detecting heart beat from tracked signals.
Initially, only Y components are selected from tracked traces and then filtered with 5th order butterworth bandpass filter to get only frequencies between [0.75Hz, 3Hz] -> (45bpm, 180bpm)
If Haar or dlib used as getting initial points, then PCA is used to change dimension of traces down to 5.
Dominant Frequency is found for each 5 signals with FFT. Then one of those dominant frequencies is evaluated as most periodic by checking total spectral power accounted for by the dominant frequency
Instead of selecting individual traces with PCA, mean of traces are calculated with assumption of each landmark point has equal movement. Then dominant frequency is found by appliying FFT to the mean signal.
- Make main with source selection and argument parsing
- Make signal_processing a module
- Output BPM into a file
- Make plotting faster
- For much better signal processing, add timestamp to every points tracked location in TrackPoints
- For live feed, first get real fps with calculating time between every frame, then use it as sampling rate