+ sridhar godavarthy july 01, 2010 defense of a masters thesis computer science and engineering...

+

Sridhar GodavarthyJuly 01, 2010

Defense of a Masters ThesisComputer Science and EngineeringUniversity of South Florida

Microexpression Spotting in Video Using Optical Strain

Microexpression Spotting in Video Using

Optical StrainSridhar Godavarthy

Examining Committee

Dmitry B. Goldgof, Ph.D. – Major Professor

Sudeep Sarkar, Ph.D. Rangachar Kasturi, Ph.D.

July 01, 2010

Defense of a Masters ThesisComputer Science and EngineeringUniversity of South Florida

+ Minutes of the presentationMicroexpressions - “micro” expressions.

Goal: Detect “interesting” sequences containing μE.

Approach: optical flow + strain thresholding.

Result: True positive detection as high as 80%.Good performance on real time videos.

Conclusion: Novel system. Scope for improvement. Need more datasets.

+

Introduction

+ Expressions

Social emotion conveyance

Non verbal

Voluntary or involuntary

6 primary expressions

5

Anger Disgust Fear Happiness Sadness Surprise

+ Microexpressions – What?

Subtle movements of the human body Observable Insufficient to convey emotion

Masking an expression

1/25th to 1/5th of a second

Almost impossible to suppress

07/01/2010

6

+ Why?

Lie Detection

Pain detection for autistic and anaesthetized patients

Social signal processing( boredom/ concentration detection)

Psychological counseling.

07/01/2010

7

+ State of the art

07/01/2010

8

Microexpression Research

Psychology Vision

Intent FACS classificationOptical FlowGabor FiltersANNsRule Based

+ Objective

Design a preprocessing system that

Spots microexpressions.

Handles small translational and rotational

motion

Improves performance of existing systems

Greater weight to true positives.

07/01/2010

9

+Some Fundamentals

Optic Flow : Vector representation of temporal changes

Strain: Relative deformation of material (skin)

Haar Classifier / Viola-Jones Face detector Cascade of weak classifiers OpenCV implementation Uses Haar rectangular features

+ Brief overview of algorithm ( Main Idea)Skin deforms during an expression.

Deformation peaks at peak of expression

Duration of increased strain corresponds to microexpression

07/01/2010

11

~22 frames

~5 frames

Peak DetectionThresholding

Frames

Str

ain

Mag

nitu

de

Macro Expression

Micro Expression

+

Algorithm

+System Flow

Split Frames Face Detection & Alignment Optical Flow

Strain MapSplit into regionsStrain patterns and

Thresholding

+ Face detection

Viola-Jones face detector-OpenCV implementation

+ Face Alignment: Rotation

07/01/2010

15

+System Flow

Split Frames Face Detection & Alignment Optical Flow

Strain MapSplit into regionsStrain patterns and

Thresholding

+ Optical flow

07/01/2010

17

• Black and Anandan• Dense

MJ Black’s Matlab imlementation of OF

+System Flow

Split Frames Face Detection & Alignment Optical Flow

Strain MapSplit into regionsStrain patterns and

Thresholding

+ Facial strain

07/01/2010

19

+System Flow

Split Frames Face Detection & Alignment Optical Flow

Strain MapSplit into regionsStrain patterns and

Thresholding

+ Region Splitting

forehead(fh)

right cheek (rc)

left cheek (lc)

right mouth(rm)

left mouth(lm)

right eye(re)

below mouth(bm)

AUs not covered• Blink• Close eyes• Neck tightening• Nostril flare

Automated. Manual intervention if classifier fails

+

Datasets & Results

+Datasets

USF: IRB Canal9: EULA Found Videos: Fair use act

USF(100)

Canal9 (24) Found (4)

+Threshold Determination

Sl. No.Threshold as percentage of

peak strain% True positives % False positives

175 31.8 0

250 50 0

335 77.2 22.7

430 54.5 36.4

525 13.6 0

07/01/2010

24

75 50 35 30 250

10

20

30

40

50

60

70

80

90

100

Threshold Selection

% Peak Strain

% T

rue

Po

sit

ive

+Thresholded Strain Maps – Sample 1 / 3

07/01/2010

25

+Thresholded Strain Maps – Sample 2 / 3

07/01/2010

26

+Thresholded Strain Maps – Sample 3 / 3

07/01/2010

27

False Positive ( Indicative only)

+Microexpression Spotting

07/01/2010

29

+Negative Test Case – Rejects Expressions

07/01/2010

30

+

Concluding Remarks

+Contributions and Conclusions

Automated thresholding

Automated alignment (Partial)

Region wise detection

Up to 80% true detection

Microexpressions with expressions are detected.

+Constraints

Constant illumination

Neutral face

Some expressions may be falsely detected

Talking

+ Future Work

Dataset CollectionReal time questioning videos

Fully automated face alignmentBy matching optical flow vectors

Automatic identification of neutral face

Automatic portioning of facesAnthropomorphic landmark identification

07/01/2010

34

+Related Publications

Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S., "Towards macro- and micro-expression spotting in video using strain patterns," Workshop on Applications of Computer Vision, 2009 pp:1-6

07/01/2010

35

+ Other Publications by author Candamo, J., Kasturi, R., Goldgof, D., Godavarthy, S., "Detecting Wires in

Cluttered Urban Scenes Using a Gaussian Model, " to appear in Proceedings of International Conference on Pattern Recognition(ICPR 2010), Turkey, 2010

Godavarthy, S., Roomi, M. Md., “Adaptive Contrast Based Unsharp Masking,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002

Godavarthy, S., Pandian, A., Roomi, M. Md., “Histogram Equalization by Measure of Enhancement,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002

Godavarthy, S., Shankar, A., Roomi, M. Md., “Adaptive Watermarking-a FFT Approach”, Proceedings of International Conference on Advances in Telecommunication and Information Technology "Asia - Pacific Telecom 2000" (14th, 15th December 2000), Vellore

07/01/2010

36

+Dr. Ekman on A-Rod

http://www.nytimes.com/2009/02/15/weekinreview/15marsh.html

+

THANK YOU

38

+Index Presentation:

Minutes

System Flow

Thresholding

Sample Strain Maps

Results

Negative Test Case

Conclusion and Future Work

Additional Slides

Evolutionary psychology

Detailed Flow Chart

Optical Flow

Elasticity and Strain

FACS

OF Vs OS

Dataset Details

+

Additional Slides

+

Study of everything we discussed until now

The child of ONE man - Paul Ekman. Over thirty years of research One of the world’s leading experts on lying. About 2 dozen books and innumerable articles Developed FACS Scientific Advisor to “Lie to Me” Co creator of Microexpression Training Tool (METTx)

Evolutionary Psychology

+Flow Chart

07/01/2010

42

Split into individual frames

Detect and Crop face

Translate/Rotate to align with neutral mage

Calculate Robust Optical Flow

Calculate Optical Strain

Displacement Vectors

Split into Regions

Compute and normalize strain/region

Strain per pixel

Eight regions or less depending on visibility

Thresholding for magnitude, duration and spatial locality

Microexpression

Input video sequence

Viola-Jones Face Detector

Haar Cascade, Skin Detection

Finite Difference Method

Black and Anandan Method

Determine threshold for Strain Magnitude

Threshold

+ Add: OFMotion Estimation: Optical Flow Method

Reflects the changes in the image due to motion

Computation is based on the following assumptions: observed brightness of any object point is constant over time nearby points in the image plane move in a similar manner

Minimization problem: (brightness const.) (smoothness const.)

Robust estimation framework (Black and Anandan, 1996) Recast the least squared formulations with a different error-norm function instead of

quadratic Coarse-to-fine strategy

Construct a pyramid of spatially filtered and sub-sampled images Compute flow values at lowest resolution and project to next level in the

pyramid

43

)()(),( 222222yxyxtyx vvuufvfufyxE

10/29/2009

+

• Def: Optical Flow is the apparent motion of brightness patterns in the image

• Ideally, same as the motion field

• Have to be careful: apparent motion can be caused by lighting changes without any actual motion

Optical Flow

• Key assumptions• Brightness constancy: projection of the same

point looks the same in every frame• Small motion: points do not move very far• Spatial coherence: points move like their

neighbors

Elasticity

Different materials have different elasticity

Elasticity can be modeled

strain

stressElasticity Known

Calculate

+

What is Facial Strain? Strain on soft tissue when expressions are made. Anatomical method Uses a pair of frames to measure deformation

Facial Strain

+

Finite Difference Method

Compute spatial derivatives from discrete points. Forward Difference Method Central Difference Method Richardson extrapolation

Strain Measurement

+Thresholding

Threshold Strain Maps to segment out μE

07/01/2010

48

+The Facial Action Coding System (FACS)

Coding of human expressions

Observational and Anatomical

32 Action Units and 14 Action Descriptors

Encode any possible [facial] expression.

Also used for facial expression simulations

+ Name Method Type Content

Advantages Disadvantages

FAST O T S Early method Only negative emotions

FACS O A C All muscles Allows for

discovery

-

MAX O T S Faster performance

Only pre defined configurations.

EMG Obt A C Muscular activity invisible to naked eye

Interference from nearby muscles

EMFACS O T S Faster performance.

Only certain emotion expressions.

O - Observational Obt - Obtrusive A – Anatomical T – Theoretical S – Selective C – Comprehensive

+FACS examples

07/01/2010

51

+Why Optical Strain?

07/01/2010

52

+ Datasets

Dataset Name

No. of Sequence

s

Approximate Duration

per sequence(s)

Microexpressions

per sequenc

e

Total Resolution

USF – feigned 12 140 8 96 SD / HD

USF – questioning 4 65 1 4 HD

Canal9 dataset 6 300-400 4 24 HD

Found videos 3 30-40 1 4 Very Low

07/01/2010

53

+Index Presentation:

Minutes

System Flow

Thresholding

Sample Strain Maps

Results

Negative Test Case

Conclusion and Future Work

Additional Slides

Evolutionary psychology

Detailed Flow Chart

Optical Flow

Elasticity and Strain

FACS

OF Vs OS

Dataset Details