+ sridhar godavarthy july 01, 2010 defense of a masters thesis computer science and engineering...
TRANSCRIPT
+
Sridhar GodavarthyJuly 01, 2010
Defense of a Masters ThesisComputer Science and EngineeringUniversity of South Florida
Microexpression Spotting in Video Using Optical Strain
Microexpression Spotting in Video Using
Optical StrainSridhar Godavarthy
Examining Committee
Dmitry B. Goldgof, Ph.D. – Major Professor
Sudeep Sarkar, Ph.D. Rangachar Kasturi, Ph.D.
July 01, 2010
Defense of a Masters ThesisComputer Science and EngineeringUniversity of South Florida
+ Minutes of the presentationMicroexpressions - “micro” expressions.
Goal: Detect “interesting” sequences containing μE.
Approach: optical flow + strain thresholding.
Result: True positive detection as high as 80%.Good performance on real time videos.
Conclusion: Novel system. Scope for improvement. Need more datasets.
+
Introduction
+ Expressions
Social emotion conveyance
Non verbal
Voluntary or involuntary
6 primary expressions
5
Anger Disgust Fear Happiness Sadness Surprise
+ Microexpressions – What?
Subtle movements of the human body Observable Insufficient to convey emotion
Masking an expression
1/25th to 1/5th of a second
Almost impossible to suppress
07/01/2010
6
+ Why?
Lie Detection
Pain detection for autistic and anaesthetized patients
Social signal processing( boredom/ concentration detection)
Psychological counseling.
07/01/2010
7
+ State of the art
07/01/2010
8
Microexpression Research
Psychology Vision
Intent FACS classificationOptical FlowGabor FiltersANNsRule Based
+ Objective
Design a preprocessing system that
Spots microexpressions.
Handles small translational and rotational
motion
Improves performance of existing systems
Greater weight to true positives.
07/01/2010
9
+Some Fundamentals
Optic Flow : Vector representation of temporal changes
Strain: Relative deformation of material (skin)
Haar Classifier / Viola-Jones Face detector Cascade of weak classifiers OpenCV implementation Uses Haar rectangular features
+ Brief overview of algorithm ( Main Idea)Skin deforms during an expression.
Deformation peaks at peak of expression
Duration of increased strain corresponds to microexpression
07/01/2010
11
~22 frames
~5 frames
Peak DetectionThresholding
Frames
Str
ain
Mag
nitu
de
Macro Expression
Micro Expression
+
Algorithm
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
+ Face detection
Viola-Jones face detector-OpenCV implementation
+ Face Alignment: Rotation
07/01/2010
15
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
+ Optical flow
07/01/2010
17
• Black and Anandan• Dense
MJ Black’s Matlab imlementation of OF
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
+ Facial strain
07/01/2010
19
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
+ Region Splitting
forehead(fh)
right cheek (rc)
left cheek (lc)
right mouth(rm)
left mouth(lm)
right eye(re)
below mouth(bm)
AUs not covered• Blink• Close eyes• Neck tightening• Nostril flare
Automated. Manual intervention if classifier fails
+
Datasets & Results
+Datasets
USF: IRB Canal9: EULA Found Videos: Fair use act
USF(100)
Canal9 (24) Found (4)
+Threshold Determination
Sl. No.Threshold as percentage of
peak strain% True positives % False positives
175 31.8 0
250 50 0
335 77.2 22.7
430 54.5 36.4
525 13.6 0
07/01/2010
24
75 50 35 30 250
10
20
30
40
50
60
70
80
90
100
Threshold Selection
% Peak Strain
% T
rue
Po
sit
ive
+Thresholded Strain Maps – Sample 1 / 3
07/01/2010
25
+Thresholded Strain Maps – Sample 2 / 3
07/01/2010
26
+Thresholded Strain Maps – Sample 3 / 3
07/01/2010
27
False Positive ( Indicative only)
+Microexpression Spotting
07/01/2010
29
+Negative Test Case – Rejects Expressions
07/01/2010
30
+
Concluding Remarks
+Contributions and Conclusions
Automated thresholding
Automated alignment (Partial)
Region wise detection
Up to 80% true detection
Microexpressions with expressions are detected.
+Constraints
Constant illumination
Neutral face
Some expressions may be falsely detected
Talking
+ Future Work
Dataset CollectionReal time questioning videos
Fully automated face alignmentBy matching optical flow vectors
Automatic identification of neutral face
Automatic portioning of facesAnthropomorphic landmark identification
07/01/2010
34
+Related Publications
Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S., "Towards macro- and micro-expression spotting in video using strain patterns," Workshop on Applications of Computer Vision, 2009 pp:1-6
07/01/2010
35
+ Other Publications by author Candamo, J., Kasturi, R., Goldgof, D., Godavarthy, S., "Detecting Wires in
Cluttered Urban Scenes Using a Gaussian Model, " to appear in Proceedings of International Conference on Pattern Recognition(ICPR 2010), Turkey, 2010
Godavarthy, S., Roomi, M. Md., “Adaptive Contrast Based Unsharp Masking,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002
Godavarthy, S., Pandian, A., Roomi, M. Md., “Histogram Equalization by Measure of Enhancement,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002
Godavarthy, S., Shankar, A., Roomi, M. Md., “Adaptive Watermarking-a FFT Approach”, Proceedings of International Conference on Advances in Telecommunication and Information Technology "Asia - Pacific Telecom 2000" (14th, 15th December 2000), Vellore
07/01/2010
36
+Dr. Ekman on A-Rod
http://www.nytimes.com/2009/02/15/weekinreview/15marsh.html
+
THANK YOU
38
+Index Presentation:
Minutes
System Flow
Thresholding
Sample Strain Maps
Results
Negative Test Case
Conclusion and Future Work
Additional Slides
Evolutionary psychology
Detailed Flow Chart
Optical Flow
Elasticity and Strain
FACS
OF Vs OS
Dataset Details
+
Additional Slides
+
Study of everything we discussed until now
The child of ONE man - Paul Ekman. Over thirty years of research One of the world’s leading experts on lying. About 2 dozen books and innumerable articles Developed FACS Scientific Advisor to “Lie to Me” Co creator of Microexpression Training Tool (METTx)
Evolutionary Psychology
+Flow Chart
07/01/2010
42
Split into individual frames
Detect and Crop face
Translate/Rotate to align with neutral mage
Calculate Robust Optical Flow
Calculate Optical Strain
Displacement Vectors
Split into Regions
Compute and normalize strain/region
Strain per pixel
Eight regions or less depending on visibility
Thresholding for magnitude, duration and spatial locality
Microexpression
Input video sequence
Viola-Jones Face Detector
Haar Cascade, Skin Detection
Finite Difference Method
Black and Anandan Method
Determine threshold for Strain Magnitude
Threshold
+ Add: OFMotion Estimation: Optical Flow Method
Reflects the changes in the image due to motion
Computation is based on the following assumptions: observed brightness of any object point is constant over time nearby points in the image plane move in a similar manner
Minimization problem: (brightness const.) (smoothness const.)
Robust estimation framework (Black and Anandan, 1996) Recast the least squared formulations with a different error-norm function instead of
quadratic Coarse-to-fine strategy
Construct a pyramid of spatially filtered and sub-sampled images Compute flow values at lowest resolution and project to next level in the
pyramid
43
)()(),( 222222yxyxtyx vvuufvfufyxE
10/29/2009
+
• Def: Optical Flow is the apparent motion of brightness patterns in the image
• Ideally, same as the motion field
• Have to be careful: apparent motion can be caused by lighting changes without any actual motion
Optical Flow
• Key assumptions• Brightness constancy: projection of the same
point looks the same in every frame• Small motion: points do not move very far• Spatial coherence: points move like their
neighbors
Elasticity
Different materials have different elasticity
Elasticity can be modeled
strain
stressElasticity Known
Calculate
+
What is Facial Strain? Strain on soft tissue when expressions are made. Anatomical method Uses a pair of frames to measure deformation
Facial Strain
+
Finite Difference Method
Compute spatial derivatives from discrete points. Forward Difference Method Central Difference Method Richardson extrapolation
Strain Measurement
+Thresholding
Threshold Strain Maps to segment out μE
07/01/2010
48
+The Facial Action Coding System (FACS)
Coding of human expressions
Observational and Anatomical
32 Action Units and 14 Action Descriptors
Encode any possible [facial] expression.
Also used for facial expression simulations
+ Name Method Type Content
Advantages Disadvantages
FAST O T S Early method Only negative emotions
FACS O A C All muscles Allows for
discovery
-
MAX O T S Faster performance
Only pre defined configurations.
EMG Obt A C Muscular activity invisible to naked eye
Interference from nearby muscles
EMFACS O T S Faster performance.
Only certain emotion expressions.
O - Observational Obt - Obtrusive A – Anatomical T – Theoretical S – Selective C – Comprehensive
+FACS examples
07/01/2010
51
+Why Optical Strain?
07/01/2010
52
+ Datasets
Dataset Name
No. of Sequence
s
Approximate Duration
per sequence(s)
Microexpressions
per sequenc
e
Total Resolution
USF – feigned 12 140 8 96 SD / HD
USF – questioning 4 65 1 4 HD
Canal9 dataset 6 300-400 4 24 HD
Found videos 3 30-40 1 4 Very Low
07/01/2010
53
+Index Presentation:
Minutes
System Flow
Thresholding
Sample Strain Maps
Results
Negative Test Case
Conclusion and Future Work
Additional Slides
Evolutionary psychology
Detailed Flow Chart
Optical Flow
Elasticity and Strain
FACS
OF Vs OS
Dataset Details