Skip to content

Latest commit

 

History

History
118 lines (68 loc) · 7.45 KB

README.md

File metadata and controls

118 lines (68 loc) · 7.45 KB

Blink Detection Experiments:

2019-2020 Spring CSE4097 - Graduation Project I

3 different blink detection methods are examined in the scope of graduation thesis as a side project. We were planning to uses these methods to extract blink-based features, but we gave up on these later.

  1. simple thresholding: Eye-aspect-ratio based model with constant thresholds which defines a proper blink behaviour.

  2. adaptive model: Same approach above with adaptive thresholds.

  3. machine learning model: Completely different experiment with a machine learning approach.


Online Notebooks:

  1. Simple model

  2. Adaptive model

  3. a. ML model part-1

    b. ML model part-2

  4. Comparison of models

  5. a. Utility script-1

    b. Utility script-2

    c. Utility script-3


Repo Content and Implementation Steps:

  • Basic model is an implementation of paper by Kazemi and Sullivan (2014). It uses Dlib's "get_frontal_face_detector" (HoG and SVM) to detect faces and "shape_predictor" (ensemble of regression trees) to detect facial landmarks. Then it uses EAR formula to get eye openness.

  • After calculating EAR values, blinks are detected by using 3 constant thresholds:

    # eye aspect ratio to indicate blink
    EAR_THRESHOLD = 0.21
    
    # number of consecutive frames the eye must be below the threshold
    EAR_CONSEC_FRAMES = 3 
    
    # how many frames we should skip at the beggining
    SKIP_FIRST_FRAMES = 0 
    
  • Building a pipeline to read videos and annotations of TalkingFace dataset, to evaluate the simple model on it and to write / to read output files.

  • Analyzing blink detection problem with timeseries approach. The goal is to make 3 thresholds adaptive to any situation.

  • To make the threshold SKIP_FIRST_FRAMES adaptive, converging slope of linear fitting line is used.

  • Instead of using a constant value for EAR_THRESHOLD, outlier detection is used. 3 different methods IQR, confidence intervals and limiting z values are experimented.

  • To overcome the issue with false blinks while yawning and smiling, Error of EAR-EWMA (exponentially weighted moving average) is used insted of direct usage of EAR values in outlier detection.

  • For the third threshold EAR_CONSEC_FRAMES, the relation between significant values of PACF plot and blink durations, is examined.

  • Stationarity analysis, Dickey-Fuller test, rolling mean, exponentially decaying and time-shifting and confidence intervals are used to build a handcrafted function to estimate significant values of PACF.

  • Another method, grid search with ARIMA, is tested. Best parameters of ARIMA is found then p value is chosen as significant value of PACF.

  • Other experiments on exploring RSI (Relative Strength Indicator), CasualImpact.


  • After all of these experiments, the most efficient adaptive model is decided as a pipeline of
    EWMA + outlier detection with z values + guessing significant points of PACF.

  • For this part Eyeblink8 dataset is processed to get EAR values and annotations to train ML models in classification phase.
  • Comparison of 4 models: simple model, adaptive model, ml model, ml model from RLDD.
  • Utility functions based on "eye_blink_detection_1_simple_model.ipynb".
  • Utility functions based on "eye_blink_detection_2_adaptive_model.ipynb".
  • Utility functions based on "eye_blink_detection_3_ml_model_part1" and "eye_blink_detection_3_ml_model_part2".