19 cv mil_temporal_models

Computer vision: models, learning and inference

Chapter 19 Temporal models

Please send errata to s.prince@cs.ucl.ac.uk

Goal

To track object state from frame to frame in a video

Difficulties:

• Clutter (data association)• One image may not be enough to fully define state• Relationship between frames may be complicated

3

Structure

3Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Temporal models• Kalman filter• Extended Kalman filter• Unscented Kalman filter• Particle filters• Applications

4

Temporal Models

• Consider an evolving system• Represented by an unknown vector, w• This is termed the state• Examples:– 2D Position of tracked object in image– 3D Pose of tracked object in world– Joint positions of articulated model

• OUR GOAL: To compute the marginal posterior distribution over w at time t.

4Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

5

Estimating State

Two contributions to estimating the state:1. A set of measurements xt, which provide

information about the state wt at time t. This is a generative model: the measurements are derived from the state using a known probability relation Pr(xt|w1…wT)

2. A time series model, which says something about the expected way that the system will evolve e.g., Pr(wt|w1...wt-1,wt+1…wT)

5Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Only the immediate past matters (Markov)

– the probability of the state at time t is conditionally independent of states at times 1...t-2 given the state at time t-1.

• Measurements depend on only the current state

– the likelihood of the measurements at time t is conditionally independent of all of the other measurements and the states at times 1...t-1, t+1..t given the state at time t.

Assumptions

6Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Graphical Model

World states

Measurements

7Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Recursive EstimationTime 1

Time 2

Time tfrom

temporal model

8Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Computing the prior (time evolution)

Each time, the prior is based on the Chapman-Kolmogorov equation

Prior at time t Temporal model Posterior at time t-1

9Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Summary

Temporal Evolution

Measurement Update

Alternate between:

Temporal model

Measurement model

10Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

11

Structure

11Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Temporal models• Kalman filter• Extended Kalman filter• Unscented Kalman filter• Particle filters• Applications

Kalman FilterThe Kalman filter is just a special case of this type of recursive estimation procedure.

Temporal model and measurement model carefully chosen so that if the posterior at time t-1 was Gaussian then the

• prior at time t will be Gaussian• posterior at time t will be Gaussian

The Kalman filter equations are rules for updating the means and covariances of these Gaussians

12Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

The Kalman Filter

Previous time step Prediction

Measurement likelihood Combination13Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Kalman Filter Definition

Time evolution equation

Measurement equationState transition matrix Additive Gaussian noise

Additive Gaussian noiseRelates state and measurement

14Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Kalman Filter Definition

Time evolution equation

Measurment equationState transition matrix Additive Gaussian noise

Additive Gaussian noiseRelates state and measurement

15Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Temporal evolution

16Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Measurement incorporation

17Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Kalman Filter

This is not the usual way these equations are presented.

Part of the reason for this is the size of the inverses: f is usually landscape and so fTf is inefficient

Define Kalman gain:

18Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Mean Term

Using Matrix inversion relations:

19Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Covariance TermKalman Filter

Using Matrix inversion relations:

20Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Final Kalman Filter Equation

Innovation (difference betweenactual and predicted measurements

Prior variance minus a term due to information from measurement

21Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Kalman Filter SummaryTime evolution equation

Measurement equation

Inference

22Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Kalman Filter Example 1

23Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Kalman Filter Example 2

Alternates:

24Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

25

Smoothing• Estimates depend only on measurements up to the current point in time.• Sometimes want to estimate state based on future measurements as well

Fixed Lag Smoother:

This is an on-line scheme in which the optimal estimate for a state at time t -t is calculated based on measurements up to time t, where t is the time lag. i.e. we wish to calculate Pr(wt-t |x1 . . .xt ).

Fixed Interval Smoother:

We have a fixed time interval of measurements and want to calculate the optimal state estimate based on all of these measurements. In other words, instead of calculating Pr(wt |x1 . . .xt ) we now estimate Pr(wt |x1 . . .xT) where T is the total length of the interval.

25Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

26

Fixed lag smoother

26Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

State evolution equation

Measurement equation

Estimate delayed by t

Fixed-lag Kalman Smoothing

27Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Fixed interval smoothing

28Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Backward set of recursions

where

Equivalent to belief propagation / forward-backward algorithm

Temporal Models

29Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Problems with the Kalman filter

• Requires linear temporal and measurement equations

• Represents result as a normal distribution: what if the posterior is genuinely multi-modal?

30Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

31

Structure

31Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Temporal models• Kalman filter• Extended Kalman filter• Unscented Kalman filter• Particle filters• Applications

Roadmap

32Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Extended Kalman FilterAllows non-linear measurement and temporal equations

Key idea: take Taylor expansion and treat as locally linear

33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

JacobiansBased on Jacobians matrices of derivatives

34Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Extended Kalman Filter Equations

35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Extended Kalman Filter

36Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Problems with EKF

37Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

38

Structure

38Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Temporal models• Kalman filter• Extended Kalman filter• Unscented Kalman filter• Particle filters• Applications

Unscented Kalman Filter

Key ideas:

• Approximate distribution as a sum of weighted particles with correct mean and covariance

• Pass particles through non-linear function of the form

• Compute mean and covariance of transformed variables

39Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Unscented Kalman Filter

Choose so that

40Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Approximate with particles:

One possible scheme

With:

41Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Reconstitution

42Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Unscented Kalman Filter

43Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Measurement incorportation

44Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Measurement incorporation works in a similar way:Approximate predicted distribution by set of particles

Particles chosen so that mean and covariance the same

Measurement incorportation

45Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Measurement update equations:

Kalman gain now computed from particles:

Pass particles through measurement equationand recompute mean and variance:

Problems with UKF

46Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

47

Structure

47Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Temporal models• Kalman filter• Extended Kalman filter• Unscented Kalman filter• Particle filters• Applications

Particle filters

48Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Key idea:

• Represent probability distribution as a set of weighted particles

Advantages and disadvantages:

+ Can represent non-Gaussian multimodal densities+ No need for data association- Expensive

Condensation Algorithm

49Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Stage 1: Resample from weighted particles according to their weight to get unweighted particles

Condensation Algorithm

50Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Stage 2: Pass unweighted samples through temporal model and add noise

Condensation Algorithm

51Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Stage 3: Weight samples by measurement density

Data Association

52Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

53

Structure

53Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Temporal models• Kalman filter• Extended Kalman filter• Unscented Kalman filter• Particle filters• Applications

54

Tracking pedestrians

54Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

5555Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Tracking contour in clutter

56

Simultaneous localization and mapping

56Computer vision: models, learning and inference. ©2011 Simon J.D. Prince