signals relevant to counter-terror

M. Frank & D. Metaxas - Rutgers HMS Symposium 2003

Computer vision approaches to identifying Computer vision approaches to identifying people and possible malfeasant behavior. people and possible malfeasant behavior.

Dimitris N. Metaxas Mark G. FrankDimitris N. Metaxas Mark G. Frank Director, Computational Biomedicine School of Communication, Imaging & Modeling Center Information & Library Studies

with special thanks to: Paul Ekman, UCSFUCSF;Sinuk Kang, Amy Marie Keller, Anastacia Kurylo, Maggie Herbasz, Belida Uckun, RutgersRutgers;Jeff Cohn, PittPitt; Takeo Kanade, CMUCMU; Javier Movellan & Marni Bartlett, UCSDUCSD.David Dinges, UPENN

Also thanks to: Office of Naval Research, National Science Foundation (ITR program), AFOSR, DARPA


Signals relevant to counter-terror.Signals relevant to counter-terror.

• Identification of bad guysIdentification of bad guys• Changes in gait with loads as small as 1 kgChanges in gait with loads as small as 1 kg• Anger prior to imminent attackAnger prior to imminent attack• Fear/distress when lyingFear/distress when lying


Take a closer look…Take a closer look…

Kim Philby, 1960’sKim Philby, 1960’s (Frank & Ekman, 2003)(Frank & Ekman, 2003)


How lies are betrayed.How lies are betrayed.

LieLie

Cognitive cluesCognitive clues

- Contradictory statements- Contradictory statements- HesitationsHesitations

-Speech errorsSpeech errors-Reduced illustratorsReduced illustrators

-Contradictory emblemsContradictory emblems-Reduced detailReduced detail

-Etc.Etc.

Emotional cluesEmotional clues

Lying about feelingsLying about feelings Feelings about lyingFeelings about lying

-Look for -Look for reliablereliable

signs of signs of emotionemotion

-Duping delight-Duping delight-GuiltGuilt

-DetectionDetectionapprehensionapprehension


The Facial Action Coding System The Facial Action Coding System (FACS)(FACS)

Ekman & Friesen, 1978

Action code: 1, 2, 4, 5, 7, 20, 26

1 Inner brow raise2 Outer brow raise4 Brow lower5 Upper lid raise7 Lid tighten20 Lip stretch26 Jaw drop

46 Action Units


Challenges facing behavioral science:Challenges facing behavioral science:

• Advantages:Advantages:- reliably identify people & behaviors - non-obtrusive- non-inferential, allows for discovery

• Disadvantage:Disadvantage:- laborious - mistaken identity via cognitive capacity, disguise, etc

• SolutionSolution- automatic computer vision techniques


Gait recognition

• Identify people from the way they walk

• Important for surveillance and intrusion detection

• What are good features for identifying a person?– i.e., what features are person-specific?


Background

Sagittal plane - divides body into left and right halves

Limb segment - a vector between two sites on a particular limb


Elevation Angles


The trajectories of the sagittal elevation angles are invariant across different subjects.

As a consequence, person-independent gait recognition will require less training data.

(Borghese et al., 1996)


The cyclogram

• Elevation angles trace curve in a 4D space

• Curve is called “cyclogram”

• Cyclogram lies in a 2D plane– Well, almost

• Hypothesis: deviation of cyclogram from plane is person-specific


Cyclogram example

Curve is cyclogram projected into best-fit plane

Green points are real points of cyclogram

Red lines trace the deviation of points from plane (exaggerated scale)


Cyclogram sequence

• Deviation from cyclogram plane can be represented as a sequence

• e.g., CCCGTTTTATATTTTTAAAAGCCGGTAAATTAGGGG

• Compare sequences between people via longest common subsequence (LCS) matching– Well-known dynamic programming algorithm,

used in computational biology


Examples of People Detection


Examples of Gait Analysis


Face: Tracking: Stress Recognition

• Identify which Facial Features (space and time) are important to recognize stress

• Assymetries and Movements around the mouth and eyebrows


Slope = Asymmetry

A horizontal line would indicate no asymmetry. This facial expression, however, is generally slanted upward and to the left.


Plots of high and low stress

Expression of high stress in form ofasymmetrical facial expression

time

asym

met

ry


In contrast, low stress

time

asym

met

ry


Some more high stress from different subjects


Face: Tracking: Stress Recognition


Subtle brow changes important.Subtle brow changes important.


Technical Challenges.Technical Challenges.

• PosePose• Head motionHead motion• Occlusion from glasses, facial hair, rotation, Occlusion from glasses, facial hair, rotation,

handshands• TalkingTalking• Video qualityVideo quality• Frame rate (blinks)Frame rate (blinks)


Conclusions.Conclusions.

• There are reliable means to identify people as There are reliable means to identify people as well as behaviors associated with deception and well as behaviors associated with deception and hostile intent.hostile intent.

• We can detect these behaviors. We can detect these behaviors.

• We can represent them digitally.We can represent them digitally.

• Can this make us more secure?Can this make us more secure?

signals relevant to counter-terror

Documents

metaxas mark

frank director

cyclogram plane

deviation of cyclogram

cyclogram examplecurve

cyclogram sequencedeviation

deviation of points

paul ekman