![Page 1: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/1.jpg)
Object recognition and computer vision 2009/2010Lecture 11 December 15Lecture 11, December 15
Motion and Human Motion and Human ActionsActionsIvan Laptev
[email protected] projet WILLOW ENS/INRIA/CNRS UMR 8548Equipe-projet WILLOW, ENS/INRIA/CNRS UMR 8548
Laboratoire d’Informatique, Ecole Normale Supérieure, Paris
![Page 2: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/2.jpg)
![Page 3: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/3.jpg)
Computer vision grand challenge: Computer vision grand challenge: Vid d t diVid d t diVideo understandingVideo understanding
personindoors
house
outdoorsoutdoors car enter
personcountryside exit through
outdoorsObjects:cars, glasses, people, etc…
Actions:drinking, running, door exit, car
drinking carperson
kidnappingpersone te
carcar crash
a doorbuilding enter, etc…
constraints
glass
car
personcar
street
roadfield
peopleScene categories:indoors, outdoors, t t
Geometry:Street wall fieldcandlecar
carstreet
streetcarstreet scene, etc…
Street, wall, field, stair, etc…
![Page 4: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/4.jpg)
Class overviewClass overviewClass overviewClass overviewMotivation
Historic review
Modern applications
Overview of methodsOverview of methodsRole of image measurements, priorknowledge and data association
Methods I Methods II Methods IIISilhouette methods
FG/BG separation; Motion history images,
Optical Flowgeneral OF, parametric dense OF models,
Discriminative modelsBoosted ST feature models, realistic action
y gHuman interfaces
Deformable modelsActive shape models,
articulated models
Space-time methodsST-OF models, ST
detection in movies
Local featuresDetectors, descriptors,
p
motion priors, particle filters, gesture recognition
correlation, ST self-similarity, irregular behavior
pmatching, Bag of Features represen-tations, recognition
![Page 5: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/5.jpg)
Motivation I: Artistic RepresentationMotivation I: Artistic RepresentationMotivation I: Artistic RepresentationMotivation I: Artistic RepresentationEarly studies were motivated by human representations in Arts
Da Vinci: “it is indispensable for a painter, to become totally familiar with the anatomy of nerves, bones, muscles, and sinews, such that he understands for their various motions and stresses, which sinews or which muscle causes a particular motion”
“I ask for the weight [pressure] of this man for every segment of motion when climbing those stairs, and for the weight he places on b and on c. Note the vertical line below the center of mass of this man.”
Leonardo da Vinci (1452–1519): A man going upstairs, or up a ladder.
![Page 6: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/6.jpg)
Motivation II: BiomechanicsMotivation II: BiomechanicsMotivation II: BiomechanicsMotivation II: Biomechanics
The emergence of biomechanics
Borelli applied to biology the
analytical and geometrical methods, developed by Galileo Galilei
He was the first to understand that bones serve as levers and muscles function according to mathematical
principles
His physiological studies included muscle analysis and a mathematical discussion of movements, such as running or jumping
Giovanni Alfonso Borelli (1608–1679)
g j p g
![Page 7: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/7.jpg)
Motivation III: Study of motionMotivation III: Study of motionMotivation III: Study of motionMotivation III: Study of motionEtienne-Jules Marey: (1830 1904) d(1830–1904) made Chronophotographic experiments influential for the emerging field offor the emerging field of cinematography
Eadweard Muybridge (1830–1904) invented a machine for displaying the recorded series of images. He pioneered motion pictures and applied his technique toapplied his technique to movement studies
![Page 8: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/8.jpg)
Motivation III: Study of motionMotivation III: Study of motion
Gunnar Johansson [1973] pioneered studies on the use of image
Motivation III: Study of motionMotivation III: Study of motion
Gunnar Johansson [1973] pioneered studies on the use of image sequences for a programmed human motion analysis
“Moving Light Displays” (LED) enable identification of familiar people g g p y ( ) p pand the gender and inspired many works in computer vision.
Gunnar Johansson, Perception and Psychophysics, 1973
![Page 9: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/9.jpg)
Human actions: Historic reviewHuman actions: Historic reviewHuman actions: Historic reviewHuman actions: Historic review
15th 15th centurystudies of anatomy
17th centuryemergence ofbiomechanics
19th centuryemergence of
emergence of
cinematography1973 t di f hstudies of human
motion perception
Modern computer vision
![Page 10: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/10.jpg)
Modern applicationsModern applications:: AnimationAnimationModern applicationsModern applications: : AnimationAnimation
Motion Synthesis from AnnotationsOkan Arikan, David A. Forsyth, James O'Brien, SIGGRAPH 2003
![Page 11: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/11.jpg)
Modern applicationsModern applications:: AnimationAnimationModern applicationsModern applications: : AnimationAnimation
Motion Synthesis from AnnotationsOkan Arikan, David A. Forsyth, James O'Brien, SIGGRAPH 2003
![Page 12: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/12.jpg)
Modern applications:Modern applications: Video editingVideo editingModern applications: Modern applications: Video editingVideo editing
Space-Time Video CompletionY. Wexler, E. Shechtman and M. Irani, CVPR 2004
![Page 13: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/13.jpg)
Modern applications:Modern applications: Video editingVideo editingModern applications: Modern applications: Video editingVideo editing
Space-Time Video CompletionY. Wexler, E. Shechtman and M. Irani, CVPR 2004
![Page 14: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/14.jpg)
Modern applications:Modern applications: Video editingVideo editingModern applications: Modern applications: Video editingVideo editing
Recognizing Action at a DistanceAlexei A. Efros, Alexander C. Berg, Greg Mori, Jitendra Malik, ICCV 2003
![Page 15: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/15.jpg)
Modern applications:Modern applications: Video editingVideo editingModern applications: Modern applications: Video editingVideo editing
Recognizing Action at a DistanceAlexei A. Efros, Alexander C. Berg, Greg Mori, Jitendra Malik, ICCV 2003
![Page 16: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/16.jpg)
Applications: HumanApplications: Human--Machine InterfacesMachine InterfacesApplications: HumanApplications: Human--Machine InterfacesMachine Interfaces
http://vismod.media.mit.edu/vismod/demos/kidsroom/kidsroom.html
![Page 17: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/17.jpg)
Applications: Unusual Activity DetectionApplications: Unusual Activity DetectionApplications: Unusual Activity DetectionApplications: Unusual Activity Detectione.g. for surveillancee.g. for surveillance
Detecting Irregularities in I d i VidImages and in Video
Boimana & Irani, ICCV 2005
![Page 18: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/18.jpg)
Applications: Applications: Search & IndexingSearch & IndexingVideo search
pppp gg
Home videos: e.g.“My daughter climbing”
TV & Web: e.g. “Fight in a parlament”
Surveillance:suspicious behavior
f f
Video mining
Useful for TV production, entertainment, social studies, security,
Auto scripting (video2text) Video mininge.g. Discover age-smoking-gender correlations now
Auto-scripting (video2text) JANE
I need a father who's a role model, not some horny geek-boy who's gonna spray his shorts whenever I bring a
correlations nowvs. 20 years ago
girlfriend home from school.(snorts)
What a lame-o. Somebody really should put him out of his misery.
![Page 19: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/19.jpg)
Applications: Video AnnotationApplications: Video Annotationppppfor video search, for video search, indexing, indexing, etc…etc…
Learning realistic human actions from moviesLaptev, Marszalek, Schmid and Rozenfeld, CVPR 2008
![Page 20: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/20.jpg)
How to recognize actions?How to recognize actions?
![Page 21: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/21.jpg)
Action understanding: Key componentsAction understanding: Key componentsAction understanding: Key componentsAction understanding: Key components
Image measurements Prior knowledge
Foreground segmentation Image
Image measurements Prior knowledge
Deformable contour modelsg Image
gradient Associationmodels
Optical flow
2D/3D body models
Local space-time features
Automatic(Semi-) Manual
Motion priorsBackground models
Space-time templates
=
result=
training annotation
SVM classifiers
![Page 22: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/22.jpg)
Foreground regions segmentationForeground regions segmentationg g gg g gImage differencing: one of the simplest ways to measure motion/change
- > C
Better Background (BG) / Foreground (FG) separation methods are available:
Modeling of color variation at each pixel with Gaussian Mixture Models (GMMs). Dominant motion estimation and compensation for sequences with moving camera
Motion layer separation for scenes with non-static backgrounds
![Page 23: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/23.jpg)
Foreground regions segmentationForeground regions segmentationg g gg g g
+ Simple and fast
Pros:
+ Gives acceptable results under restricted conditions
Cons:- Often unreliable due to shadows, low image contrast, etc.
- Requires background model => not well suited for scenes
Cons:
Requires background model => not well suited for sceneswith dynamic BG and/or motion parallax
![Page 24: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/24.jpg)
Temporal Templates of Temporal Templates of BobickBobick & Davis& Davisp pp p
Idea: summarize motion in video in aMotion History Image (MHI):
The Recognition of Human Movement Using Temporal Templates Aaron F. Bobick and James W. Davis, PAMI 2001
![Page 25: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/25.jpg)
Temporal Templates of Temporal Templates of BobickBobick & Davis& Davisp pp p
Compute MHI for each action sequence
Describe each sequence with the t l ti d l i i t
translation and scale invariant vector of 7 Hu moments
Nearest Neighbor action l ifi ti ith M h l bi
classification with Mahalanobis distance between training and test descriptors d.
![Page 26: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/26.jpg)
Aerobics Datasete ob cs se
![Page 27: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/27.jpg)
Temporal Templates: SummaryTemporal Templates: Summaryp p yp p y
Pros:
+ Simple
+ Fast
- Assumes static camera static backgroundCons:
Assumes static camera, static background
- Sensitive to segmentation errors
- Silhouettes do not capture interior motion/shape- Silhouettes do not capture interior motion/shape
Possible improvements:
Not all shapes are valid Restrict the space of admissible shapes to overcome segmentation errors
![Page 28: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/28.jpg)
Active Shape Models of Active Shape Models of CootesCootes et al.et al.ppPoint Distribution Model
Represent the shape of samples by a set Represent the shape of samples by a set of corresponding points or landmarks
Assume each shape can be represented by the linear combination of basis shapes
by the linear combination of basis shapes
such that
for mean shape
and some parameters
![Page 29: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/29.jpg)
Active Shape Models of Active Shape Models of CootesCootes et al.et al.ppBasis shapes can be found as the main modes of variation of in the training data
in the training data.
2D Example:2D Example: (each point can be thought as a shape in N-Dim pspace)
Principle Component Analysis (PCA):
Covariance matrix
Eigenvectors eigenvalues
![Page 30: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/30.jpg)
Active Shape Models of Active Shape Models of CootesCootes et al.et al.pp
Back project from shape space to image space Back-project from shape-space to image space
Three main modes of lips-shape variation:
Distribution of eigenvalues:
A small fraction of basisA small fraction of basis shapes (eigenvecors) accounts for the most of shape variation (=> landmarks arevariation (=> landmarks are redundant)
![Page 31: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/31.jpg)
Active Shape Models of Active Shape Models of CootesCootes et al.et al.pp
is orthonormal basis therefore is orthonormal basis, therefore
Given estimate of we can recover shape parameters
Projection onto the shape-space serves as a regularization Projection onto the shape space serves as a regularization
![Page 32: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/32.jpg)
Active Shape Models of Active Shape Models of CootesCootes et al.et al.How to use Active Shape Models for shape estimation?
pp
Given initial guess of model points estimate new positions using local image search, e.g. locate the closest edge point
Re-estimate shape parameters Re-estimate shape parameters
![Page 33: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/33.jpg)
Active Shape Models of Active Shape Models of CootesCootes et al.et al.
To handle translation, scale and rotation, it is useful to normalize
pp
, ,prior to shape estimation:
using similarity transformation
A simple way to estimate is to assign and to the iti d th t d d d i ti f i t imean position and the standard deviation of points in
respectively and set . For more sophisticated normalization techniques see:
Note: model parameters have to be computed using
http://www.isbe.man.ac.uk/~bim/Models/app_model.ps.gz
normalized image point coordinates
![Page 34: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/34.jpg)
Active Shape Models of Active Shape Models of CootesCootes et al.et al.ppIterative ASM alignment algorithm
1 I iti li ith th bl f d1. Initialize with the reasonable guess of and 2. Estimate from image measurements3. Re-estimate
Example: face alignment Illustration of face shape space
4. Unless converged, repeat from step 2
Example: face alignment Illustration of face shape space
Active Shape Models: Their Training and ApplicationT.F. Cootes, C.J. Taylor, D.H. Cooper, and J. Graham, CVIU 1995
![Page 35: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/35.jpg)
Active Shape Model trackingActive Shape Model trackingp gp gAim: to track ASM of time-varying shapes, e.g. human silhouettes
Impose time-continuity constraint on model parameters. For example, for shape parameters :
Gaussian noise
For similarity transformation
Update model parameters at each time frame using e.g.
More complex dynamical models possible
Kalman filter
![Page 36: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/36.jpg)
Person TrackingPerson Trackinggg
Learning flexible models from image sequencesA. Baumberg and D. Hogg, ECCV 1994
![Page 37: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/37.jpg)
Person TrackingPerson Trackinggg
Learning flexible models from image sequencesA. Baumberg and D. Hogg, ECCV 1994
![Page 38: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/38.jpg)
Active Shape Models: SummaryActive Shape Models: Summaryp yp y
Pros:
+ Shape prior helps overcoming segmentation errors
+ Fast optimizationFast optimization
+ Can handle interior/exterior dynamics
- Optimization gets trapped in local minimaCons:
- Re-initialization is problematic
Possible improvements:
Learn and apply specific motion priors for different actions
![Page 39: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/39.jpg)
Motion priorsMotion priorspp
Accurate motion models can be used both to:
Help accurate trackingRecognize actions
Goal: formulate motion models for different types of actionsand use such models for action recognition
g
and use such models for action recognition
Example:
line drawing
Drawing with 3 action modes
line drawing
scribbling
idlidle
From M. Isard and A. Blake, ICCV 1998
![Page 40: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/40.jpg)
Incorporating motion priorsIncorporating motion priorsIncorporating motion priorsIncorporating motion priors
Image measurements Data Association Prior knowledge
Foreground
Image measurements Data Association Prior knowledge
Foreground segmentation
Image gradientLearning motion
models for different actions
Particle filters
different actionsOptical Flow
![Page 41: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/41.jpg)
Bayesian TrackingBayesian TrackingBayesian TrackingBayesian TrackingGeneral framework: recognition by synthesis;
generati e modelsgenerative models; finding best explanation of the data
N t tiNotation:image data at timemodel parameters at time (e.g. shape and its dynamics)ode pa a e e s a e (e g s ape a d s dy a cs)prior density for likelihood of data for the given model configuration
We search posterior defined by the Bayes’ rule
For tracking the Markov assumption gives the prior
Temporal update rule:
![Page 42: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/42.jpg)
KalmanKalman FilteringFilteringKalmanKalman FilteringFilteringIf all probability densities are uni-modal, specifically Gussians, the posterior can be evaluated in the closed formthe posterior can be evaluated in the closed form
![Page 43: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/43.jpg)
Particle FilteringParticle FilteringParticle FilteringParticle FilteringIn reality probability densities are almost always multi-modal
![Page 44: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/44.jpg)
Particle FilteringParticle FilteringParticle FilteringParticle FilteringIn reality probability densities are almost always multi-modal
Approximate distributions with weighted particles
![Page 45: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/45.jpg)
Particle FilteringParticle FilteringParticle FilteringParticle Filtering
T ki lTracking examples:
describes leave shape describes head shapep p
CONDENSATION - conditional density propagation for visual tracking A. Blake and M. Isard IJCV 1998
![Page 46: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/46.jpg)
Learning dynamic priorLearning dynamic priorLearning dynamic priorLearning dynamic priorDynamic model: 2nd order Auto-Regressive Process
State
U d t lUpdate rule:
Model parameters:p
Learning scheme:
![Page 47: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/47.jpg)
Learning dynamic priorLearning dynamic priorLearning dynamic priorLearning dynamic prior
Learning point sequenceRandom simulation of the learned dynamical model
Statistical models of visual shape and motion A. Blake, B. Bascle, M. Isard and J. MacCormick, Phil.Trans.R.Soc. 1998
![Page 48: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/48.jpg)
Learning dynamic priorLearning dynamic priorLearning dynamic priorLearning dynamic prior
Random simulation of the learned gate dynamics
![Page 49: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/49.jpg)
Dynamics with discrete statesDynamics with discrete statesDynamics with discrete statesDynamics with discrete states
Introduce “mixed” state Continuous stateIntroduce mixed state Continuous state space (as before)
Discrete variable identifying dynamical model
Transition probability matrix
or more generally
Incorporation of the mixed-state model into a particle filter is straightforward simply use instead of and thestraightforward, simply use instead of and the corresponding update rules
![Page 50: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/50.jpg)
Dynamics with discrete statesDynamics with discrete statesDynamics with discrete statesDynamics with discrete states
Example: DrawingExample: Drawingline idle
line
scribbling
Transitionidlescribbling
Transition probability matrix
Result: simultaneously improved tracking and
line drawing
gesture recognition
scribbling
idleA mixed-state Condensation tracker with automatic model-switching
M. Isard and A. Blake, ICCV 1998
![Page 51: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/51.jpg)
Dynamics with discrete statesDynamics with discrete statesDynamics with discrete statesDynamics with discrete statesSimilar illustrated on gesture recognition in the context of a visual black-board interfaceblack board interface
A probabilistic framework for matching temporal trajectories:A probabilistic framework for matching temporal trajectories: CONDENSATION-based recognition of gestures and expressions
M.J. Black and A.D. Jepson, ECCV 1998
![Page 52: Motion and Human Motion and Human ActionsActions · kidnapping person ete car car crash ... detection in movies ... and use such models for action recognition Example: line drawing](https://reader033.vdocuments.site/reader033/viewer/2022051915/6006a6f15a346546a00d2398/html5/thumbnails/52.jpg)
So farSo farSo far…So far…
D t A i ti
Foreground
Image measurements Data Association Prior knowledge
Background modelsgsegmentation
Deformable shape
Temporal templatesParticle filters
Hu moments and Fourier descriptors
NN classifiers
Image edges
pmodels
Motion priors
Fourier descriptors