computer vision i - tracking › ~ds24 › lehre › cv1_ws_2014 › vl13.pdf · computer vision i...
TRANSCRIPT
![Page 1: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/1.jpg)
Computer Vision I -Tracking
Carsten Rother
Computer Vision I: Tracking17/02/2015
17/02/2015
[slide credits: Alex Krull]
![Page 2: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/2.jpg)
What is Tracking?
• „Tracking an object in an image sequence means continuously identifying its location when either the object or the camera are moving“ [Lepetit and Fua 2005]
• This can mean estimating in each frame:
• 2D location or window
• 6D rigid body transformation
• More complex parametric models• Active Appearance Models
• Skeleton for human pose etc.
17/02/2015 Computer Vision I: Tracking 2
![Page 3: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/3.jpg)
What is Tracking?
• „Tracking an object in an image sequence means continuously identifying its location when either the object or the camera are moving“ [Lepetit and Fua 2005]
• This can mean estimating in each frame:
• 2D location or window
• 6D rigid body transformation
• More complex parametric models• Active Appearance Models
• Skeleton for human pose etc.
17/02/2015 Computer Vision I: Tracking 3
![Page 4: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/4.jpg)
What is Tracking?
• „Tracking an object in an image sequence means continuoslyidentifying its location when either the object or the camera aremoving“ [Lepetit and Fua 2005]
• This can mean estimating in each frame:
• 2D location or window
• 6D rigid body transformation
• More complex parametric models• Active Appearance Models
• Skeleton for human pose etc.
17/02/2015 Computer Vision I: Tracking 4
![Page 5: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/5.jpg)
Tracking vs Localization
• Tracking of objects is closely related to:
• camera pose estimation in a known environment
• localization of agents (eg. Robots) in a known environment
• Reminder: SLAM has unknown location (agent, camera) and unknown environment
17/02/2015 Computer Vision I: Tracking 5
![Page 6: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/6.jpg)
Outline
• This lecture• The Bayes Filter
• Explained for localization
• Next lecture• The Particle Filter
• The Kalman Filter
• Pros and Cons
• Beyond tracking and localization
• Case study: • 6-DOF Model Based Tracking via Object Coordinate Regression
17/02/2015 Computer Vision I: Tracking 6
![Page 7: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/7.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 7
• We have:
• Probabilistic model for movement
• Probabilistic model formeasurement• Based on map of the environment
• Where is the ship?
• Using all previous andcurrent observations
![Page 8: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/8.jpg)
1tztz
1txtx
The Hidden Markov Model
17/02/2015 Computer Vision I: Tracking 8
Observations:Depth measurement
States:Positions
time
![Page 9: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/9.jpg)
The Hidden Markov Model
17/02/2015 Computer Vision I: Tracking 9
Observation Model:What is the likelihoodof an observation, given a state?
𝑝 𝑧𝑡 𝑥𝑡 =1
𝑐𝑒− 𝑧𝑡−𝑑 𝑥𝑡
2/(2𝜎2)
1tztz
1txtx
time
𝑑 𝑥𝑡 is the known true depth
![Page 10: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/10.jpg)
The Hidden Markov Model
17/02/2015 Computer Vision I: Tracking 10
Motion Model:Probability forstate transition
𝑝 𝑥𝑡+1 𝑥𝑡
1tztz
1txtx
time
![Page 11: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/11.jpg)
Probability distribution for the state given all previous and currentobservations:
• This is what we are interested in
• Eg. use maximum as current estimate
The Posterior distribution
17/02/2015 Computer Vision I: Tracking 11
tz
tx
1tz
1tx
0z
0x
...
...
𝑝 𝑥𝑡 𝑧0:𝑡
𝑥𝑡∗ = 𝑎𝑟𝑚𝑎𝑥𝑥𝑡
𝑝 𝑥𝑡 𝑧0:𝑡
![Page 12: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/12.jpg)
The Prior distribution
17/02/2015 Computer Vision I: Tracking 12
1txtx
1tz
1tx
0z
0x
...
...
𝑝 𝑥𝑡+1 𝑧0:𝑡
Probability distribution for the next state given only previousobservations:
• Intermediate step for calculating the next posterior
tz
![Page 13: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/13.jpg)
Important Distributions
17/02/2015 Computer Vision I: Tracking 13
𝑝 𝑥𝑡+1 𝑧0:𝑡
1txtx
1tz
1tx
0z
0x ...
tztz
tx
1tz
1tx
0z
0x ...
𝑝 𝑥𝑡 𝑧0:𝑡
1txtx
tz
tx
𝑝 𝑧𝑡 𝑥𝑡𝑝 𝑥𝑡+1 𝑥𝑡
Observation Model: Motion Model:
Posterior: Prior:
• Likelihood ofobservation givenstate
• Continuous Gaussianaround real depth
• Probability of newstate given old one
• Discrete Gaussian
• Probability of state given previousand current observations
• Probability of state given onlyprevious observations
![Page 14: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/14.jpg)
Probabilities - Reminder
• A random variable is denoted with 𝑥 ∈ {0,… , 𝐾}
• Discrete probability distribution: 𝑝(𝑥) satisfies 𝑥 𝑝(𝑥) = 1
• Joint distribution of two random variables: 𝑝(𝑥, 𝑧)
• Conditional distribution: 𝑝 𝑥 𝑧
• Sum rule (marginal distribution): 𝑝 𝑧 = 𝑥 𝑝(𝑥, 𝑧)
• Independent probability distribution: 𝑝 𝑥, 𝑧 = 𝑝 𝑧 𝑝 𝑥
• Product rule: 𝑝 𝑥, 𝑧 = 𝑝 𝑧 𝑥 𝑝(𝑥)
• Bayes’ rule: 𝑝 𝑥|𝑧 =𝑝(𝑧|𝑥)𝑝 𝑥
𝑝(𝑧)
17/02/2015 14Computer Vision I: Tracking
![Page 15: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/15.jpg)
Probabilities - Reminder
• Sum rule (marginal distribution): 𝑝 𝑧 = 𝑥 𝑝(𝑥, 𝑧)
• Product rule: 𝑝 𝑥, 𝑧 = 𝑝 𝑧 𝑥 𝑝(𝑥)
• Bayes’ rule: 𝑝 𝑥|𝑧 =𝑝(𝑧|𝑥)𝑝 𝑥
𝑝(𝑧)
• Sum rule (marginal distribution): 𝑝 𝑧|𝐴 = 𝑥 𝑝(𝑥, 𝑧|𝐴)
• Product rule: 𝑝 𝑥, 𝑧|𝐴 = 𝑝 𝑧 𝑥, 𝐴 𝑝(𝑥|𝐴)
• Bayes’ rule: 𝑝 𝑥|𝑧, 𝐴 =𝑝(𝑧|𝑥,𝐴)𝑝 𝑥|𝐴
𝑝(𝑧|𝐴)
17/02/2015 15Computer Vision I: Tracking
![Page 16: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/16.jpg)
Probabilities - Reminder
17/02/2015 16Computer Vision I: Tracking
𝐴 𝐵
𝐴
𝐶
𝐵
Independence
𝐴 and 𝐵 are not connected in graph
• 𝐴 does not contain informationabout 𝐵
• 𝑝 𝐴, 𝐵 = 𝑝 𝐴 𝑝 𝐵
• 𝑝 𝐴|𝐵 = 𝑝 𝐴
• 𝑝 𝐵|𝐴 = 𝑝(𝐵)
Conditional Independence
𝐴 and 𝐵 are connected only via 𝐶
• 𝐴 does not contain informationabout 𝐵 when 𝐶 is known
• 𝑝 𝐴, 𝐵|𝐶 = 𝑝 𝐴|𝐶 𝑝 𝐵|𝐶
• 𝑝 𝐴|𝐵, 𝐶 = 𝑝 𝐴|𝐶
• 𝑝 𝐵|𝐴, 𝐶 = 𝑝(𝐵|𝐶)
Two dice rolls
It rains
People have umbrellas
Tram is crowded
![Page 17: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/17.jpg)
Step by Step
17/02/2015 Computer Vision I: Tracking 17
𝑡 = 0
• Assume prior for first frame:
• Make first measurement: 𝑧0
𝑝(𝑥0)
![Page 18: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/18.jpg)
Step by Step
17/02/2015 Computer Vision I: Tracking 18
𝑡 = 0
• Calculate likelihood 𝑝(𝑧0|𝑥0) for every possible state 𝑥0:
![Page 19: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/19.jpg)
Step by Step
17/02/2015 Computer Vision I: Tracking 19
𝑡 = 0
• Calculate the posterior by• multiplying with prior• Normalizing
• Reducing uncertainty𝑝 𝑥0 𝑧0 =
𝑝(𝑥0)𝑝(𝑧0|𝑥0)
𝑥 𝑝(𝑥0 = 𝑥)𝑝(𝑧0|𝑥0 = 𝑥)
![Page 20: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/20.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 20
𝑡 = 0
𝑝 𝑥1 𝑧0 =
𝑥
𝑝 𝑥0 = 𝑥 𝑧0 𝑝 𝑥1 𝑥0 = 𝑥
• Calculate the prior by Convolutionwith motion model
• Adding uncertainty
![Page 21: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/21.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 21
𝑡 = 1
• Make new measurement: 𝑧1
![Page 22: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/22.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 22
𝑡 = 1
• Calculate likelihood 𝑝(𝑧1|𝑥1) for every possible state 𝑥1:
![Page 23: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/23.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 23
𝑡 = 1
• Calculate the posterior by• multiplying with prior• Normalizing
• Reducing uncertainty𝑝 𝑥1 𝑧0:1 =
𝑝(𝑥1|𝑧0)𝑝(𝑧1|𝑥1)
𝑥 𝑝(𝑥1 = 𝑥|𝑧0)𝑝(𝑧1|𝑥1 = 𝑥)
![Page 24: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/24.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 24
𝑡 = 1
• Calculate the prior by Convolutionwith motion model
• Adding uncertainty
𝑝 𝑥2 𝑧0:1 =
𝑥
𝑝 𝑥1 = 𝑥 𝑧0:1 𝑝(𝑥2|𝑥1 = 𝑥)
![Page 25: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/25.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 25
𝑡 = 2
• Make new measurement: 𝑧2
![Page 26: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/26.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 26
𝑡 = 2
• Calculate likelihood 𝑝(𝑧2|𝑥2) for every possible state 𝑥2:
![Page 27: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/27.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 27
𝑡 = 2
• Calculate the posterior by• multiplying with prior• Normalizing
• Reducing uncertainty𝑝 𝑥2 𝑧0:2 =
𝑝(𝑥2|𝑧0:1)𝑝(𝑧2|𝑥2)
𝑥 𝑝(𝑥2 = 𝑥|𝑧0:1)𝑝(𝑧2|𝑥2 = 𝑥)
![Page 28: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/28.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 28
𝑡 = 2
![Page 29: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/29.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 29
Algorithm:
1. Make observation
2. Calculate likelihood for every position
3. Multiply with last prior and normalize
• Calculate posterior
4. Convolution with motion model
• Calculate new prior
5. Go to 1.
![Page 30: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/30.jpg)
𝑝 𝑥0 𝑧0
Calculating the Posterior
17/02/2015 Computer Vision I: Tracking 30
𝑝(𝑥0)
0z
1x0x
1z
...
=𝑝(𝑥0)𝑝(𝑧0|𝑥0)
𝑥 𝑝(𝑥0 = 𝑥)𝑝(𝑧0|𝑥0 = 𝑥)
(Multiply and normalize)Observation model
(see proof 1)
![Page 31: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/31.jpg)
Proof 1
17/02/2015 Computer Vision I: Tracking 31
![Page 32: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/32.jpg)
Calculating the next Prior
17/02/2015 Computer Vision I: Tracking 32
𝑝(𝑥0)
𝑝 𝑥0 𝑧0
𝑝 𝑥1 𝑧0
...
=
𝑥
𝑝 𝑥0 = 𝑥 𝑧0 𝑝 𝑥1 𝑥0 = 𝑥
(Convolution)
Observation model
0z
1x0x
1z
(see proof 2)
![Page 33: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/33.jpg)
Proof 2
17/02/2015 Computer Vision I: Tracking 33
![Page 34: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/34.jpg)
Calculating the Posterior
17/02/2015 Computer Vision I: Tracking 34
𝑝(𝑥0)
𝑝 𝑥0 𝑧0
𝑝 𝑥1 𝑧0
...
Observation model
1x0x
Observation model
𝑝 𝑥1 𝑧0:1 =𝑝(𝑥1|𝑧0)𝑝(𝑧1|𝑥1)
𝑥 𝑝(𝑥1 = 𝑥)𝑝(𝑧1|𝑥1 = 𝑥)
1z0z
![Page 35: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/35.jpg)
𝑝 𝑥𝑡 𝑧0:𝑡
Calculating the Posterior (General Case)
17/02/2015 Computer Vision I: Tracking 35
𝑝(𝑥𝑡|𝑧0:𝑡−1)
tz
1txtx
1tz
...
=𝑝(𝑥𝑡|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡)
𝑥 𝑝(𝑥𝑡 = 𝑥|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡 = 𝑥)
(Multiply and normalize)Observation model
...
(see proof 3)
![Page 36: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/36.jpg)
Proof 3
17/02/2015 Computer Vision I: Tracking 36
![Page 37: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/37.jpg)
Calculating the next Prior (General Case)
17/02/2015 Computer Vision I: Tracking 37
𝑝(𝑥𝑡|𝑧0:𝑡−1)
𝑝 𝑥𝑡 𝑧0:𝑡
𝑝 𝑥𝑡+1 𝑧0:𝑡
...
=
𝑥
𝑝 𝑥𝑡 = 𝑥 𝑧0:𝑡 𝑝(𝑥𝑡+1|𝑥𝑡 = 𝑥)
(Convolution)
Observation model
...
tz
1txtx
1tz
(see proof 4)
![Page 38: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/38.jpg)
Proof 4
17/02/2015 Computer Vision I: Tracking 38
![Page 39: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/39.jpg)
Overview Continuous Approaches
17/02/2015 Computer Vision I: Tracking 39
• How to apply it in continuous space?
• Two popular alternatives:
• Particle filter• Represent prior and posterior with samples
• Kalman Filter • Represent prior and posterior distributions as Gaussians
![Page 40: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/40.jpg)
Important Distributions (Particle Filter)
17/02/2015 Computer Vision I: Tracking 40
𝑝 𝑥𝑡+1 𝑧0:𝑡
1tx1tx
tx
1tz
1tZ
tx ...
tx1tz
1tZ
0z
0x
tz
tz ...
𝑝 𝑥𝑡 𝑧0:𝑡
tx1tz
1tx
0z
𝑝 𝑧𝑡 𝑥𝑡𝑝 𝑥𝑡+1 𝑥𝑡
Observation Model: Motion Model:
Posterior: Prior:
• Likelihood ofobservation givenstate
• Continuous Gaussianaround real depth
• Probability of newstate given old one
• Continuous Gaussian
• Probability of state given previousand current observations
• Continuous, represented as set ofSamples (Particles)
• Probability of state given onlyprevious observations
• Continuous, represented as set ofSamples (Particles)
![Page 41: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/41.jpg)
Particle Filter
17/02/2015 Computer Vision I: Tracking 41
![Page 42: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/42.jpg)
Discrete Bayes Filter vs. Particle Filter
17/02/2015 Computer Vision I: Tracking 42
• Discrete Bayes Filter:
1. Make observation
• Particle Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery sample -> weights
3. Resampling according toweights
4. Randomly move samplesaccording to motion model(Sampling)
5. Go to 1.
![Page 43: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/43.jpg)
The Bayes Filter / Convolute and Multiply
17/02/2015 Computer Vision I: Tracking 43
𝑡 = 0
• Represent continuous prior with particles 𝑥𝑡1 , … , 𝑥𝑡
𝑛
• Make measurement: 𝑧𝑡
![Page 44: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/44.jpg)
Particle Filter / Resampling
17/02/2015 Computer Vision I: Tracking 44
• Calculate weight 𝑤𝑡𝑖= 𝑝(𝑧𝑡|𝑥𝑡 = 𝑥𝑡
𝑖) for every particle 𝑥𝑡𝑖:
![Page 45: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/45.jpg)
Particle Filter / Resampling
17/02/2015 Computer Vision I: Tracking 45
𝑡 = 0
• Draw samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛 from posterior by resamplingfrom 𝑥𝑡
1 , … , 𝑥𝑡𝑛 using the weights 𝑤𝑡
𝑖
• Reducing uncertainty
0 i
i
tw
![Page 46: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/46.jpg)
Particle Filter / Resampling
17/02/2015 Computer Vision I: Tracking 46
𝑡 = 0
random
0 i
i
twrandom random
• Draw samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛 from posterior by resamplingfrom 𝑥𝑡
1 , … , 𝑥𝑡𝑛 using the weights 𝑤𝑡
𝑖
• Reducing uncertainty
![Page 47: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/47.jpg)
Particle Filter / Resampling
17/02/2015 Computer Vision I: Tracking 47
𝑡 = 0
0 i
i
tw
• Draw samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛 from posterior by resamplingfrom 𝑥𝑡
1 , … , 𝑥𝑡𝑛 using the weights 𝑤𝑡
𝑖
• Reducing uncertainty
![Page 48: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/48.jpg)
Particle Filter / Resampling
17/02/2015 Computer Vision I: Tracking 48
Why is this allowed?
Resampling is like multiplication andnormalization:
• Sample density in posterior dependslinearly on
• Density of prior samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛
• Likelihood 𝑤𝑡𝑖
![Page 49: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/49.jpg)
Discrete Bayes Filter vs. Particle Filter
17/02/2015 Computer Vision I: Tracking 49
• Discrete Bayes Filter:
1. Make observation
• Particle Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery sample -> weights
3. Resampling according toweights
4. Randomly move samplesaccording to motion model(Sampling)
5. Go to 1.
![Page 50: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/50.jpg)
Particle Filter / Sampling
17/02/2015 Computer Vision I: Tracking 50
𝑡 = 0
• Obtain samples 𝑥𝑡+11 , … , 𝑥𝑡+1
𝑛 from the new prior by moving eachparticle according to motion model
• Adding uncertainty
![Page 51: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/51.jpg)
Particle Filter / Sampling
17/02/2015 Computer Vision I: Tracking 51
𝑝 𝐵 𝐴 = 1 𝑝 𝐵 𝐴 = 2
𝑝 𝐴 = 1 = 𝑝 𝐴 = 2 = 0.5
Why is this allowed?
• We want to sample: Bi~𝑝 𝐵
• Don‘t know 𝑝 𝐵
• We sample first: 𝐴𝑖~𝑝 𝐴
• And then 𝐵𝑖~𝑝 𝐵|𝐴 = 𝐴𝑖
1
1 2 3 4 5 61 2 3 4 5 6
𝐴 𝐵
![Page 52: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/52.jpg)
BA
Particle Filter / Sampling
17/02/2015 Computer Vision I: Tracking 52
1txtx
1tz
1tx
0z
0x ...
tz
• We want to sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑧0:𝑡
• Don‘t know 𝑝 𝑥𝑡+1 𝑧0:𝑡
Why is this allowed?
![Page 53: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/53.jpg)
Particle Filter / Sampling
17/02/2015 5317/02/2015 Computer Vision I: Tracking 53
1txtx1tx0x ...
1tz0z tz
• We want to sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑧0:𝑡
• Don‘t know 𝑝 𝑥𝑡+1 𝑧0:𝑡
• We use samples 𝑥𝑡𝑖~𝑝 𝑥𝑡 𝑧0:𝑡
Why is this allowed?
![Page 54: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/54.jpg)
Particle Filter / Sampling
17/02/2015 Computer Vision I: Tracking 54
• We want to sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑧0:𝑡
• Don‘t know 𝑝 𝑥𝑡+1 𝑧0:𝑡
• We use samples 𝑥𝑡𝑖~𝑝 𝑥𝑡 𝑧0:𝑡
• We sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑥𝑡 = 𝑥𝑡
𝑖 = 𝑝 𝑥𝑡+1 𝑥𝑡 = 𝑥𝑡𝑖 , 𝑧0:𝑡
1txtx1tx0x ...
1tz0z tz
Why is this allowed?
![Page 55: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/55.jpg)
Particle Filter (Tracking application)
• Tracking an object in a video sequence [Perez at al. 2002]
• States:
• 2D windows (x,y and size)
• Gaussian motion
• Observations:
• Color histograms
17/02/2015 Computer Vision I: Tracking 55
![Page 56: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/56.jpg)
Particle Filter (Tracking application)
• Pose tracking of an object in a kinect sequence [Krull at al. 2014]
• States:
• 6D Pose• 3D position
• 3D rotation
• Observations:
• Depth images
• Predicted Object coordinates
17/02/2015 Computer Vision I: Tracking 56
![Page 57: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/57.jpg)
Particle Filter (Summary)
• The Particle Filter implements the Bayes Filter
• Prior and posterior are represented as sample sets (particle)
• Likelihood is only evaluated at particles
• Multiplication -> weighted resampling
• Convolution -> random movement according to motion model
17/02/2015 Computer Vision I: Tracking 57
![Page 58: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/58.jpg)
Overview Continuous Approaches
• How to apply it in continuous space?
• Two popular alternatives:
• Particle filter• Represent prior and posterior with samples
• Kalman Filter • Represent prior and posterior distributions as Gaussians
17/02/2015 Computer Vision I: Tracking 58
![Page 59: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/59.jpg)
Kalman Filter
17/02/2015 Computer Vision I: Tracking 59
![Page 60: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/60.jpg)
Important Distributions (Kalman Filter)
17/02/2015 Computer Vision I: Tracking 60
𝑝 𝑥𝑡+1 𝑧0:𝑡
1tx1tx
tx
1tz
1tZ
tx ...
tx1tz
1tz
0z
0x
tz
tz ...
𝑝 𝑥𝑡 𝑧0:𝑡
tx1tz
1tx
0z
𝑝 𝑧𝑡 𝑥𝑡𝑝 𝑥𝑡+1 𝑥𝑡
Observation Model: Motion Model:
Posterior: Prior:
• Likelihood ofobservation givenstate
• Continuous Gaussianaround real position
• Probability of newstate given old one
• Continuous Gaussian
• Probability of state given previousand current observations
• Continuous, represented asGaussiasn
• Probability of state given onlyprevious observations
• Continuous, represented asGaussiasn
![Page 61: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/61.jpg)
Discrete Bayes Filter vs. Kalman Filter
17/02/2015 Computer Vision I: Tracking 61
• Discrete Bayes Filter:
1. Make observation
• Kalman Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery position in closed form
3. Multiply with last priorand normalize in closed form
4. Convolution with motionmodel in closed form
5. Go to 1.
![Page 62: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/62.jpg)
Kalman Filter
17/02/2015 Computer Vision I: Tracking 62
• Calculate likelihood 𝑝(𝑧𝑡|𝑥𝑡) as Gaussian:
![Page 63: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/63.jpg)
Kalman Filter
17/02/2015 Computer Vision I: Tracking 63
• Calculate the posterior by• Multiplying with prior• Normalizing
• Reducing uncertainty
𝑝 𝑥𝑡 𝑧0:𝑡 =𝑝(𝑥𝑡|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡)
𝑝(𝑥𝑡 = 𝑥|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡 = 𝑥) 𝑑 𝑥
• Closed form solution:
• Another Gaussian
![Page 64: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/64.jpg)
Discrete Bayes Filter vs. Kalman Filter
17/02/2015 Computer Vision I: Tracking 64
• Discrete Bayes Filter:
1. Make observation
• Kalman Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery position in closed form
3. Multiply with last priorand normalize in closed form
4. Convolution with motionmodel in closed form
5. Go to 1.
![Page 65: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/65.jpg)
Kalman Filter
17/02/2015 Computer Vision I: Tracking 65
• Calculate the prior byConvolution with motion model
• Adding uncertainty
𝑝 𝑥𝑡+1 𝑧0:𝑡 = 𝑝 𝑥𝑡 = 𝑥 𝑧0:𝑡 𝑝 𝑥𝑡+1 𝑥𝑡 = 𝑥 𝑑 𝑥
• Closed form solution:
• Another Gaussian
![Page 66: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/66.jpg)
Pros and Cons
17/02/2015 Computer Vision I: Tracking 66
Particle Filter:
Observation model can be anything
Kalman Filter:
Multimodal
Likelihood calculation per particlecan be expensive
Easy to implement and parallelize
Problematic in high dimensional statespace (many particles required)
Motion model can be anything
Observation model: linear transformation of stateplus Gaussian noise
Unimodal
Motion model: linear transformation of last stateplus Gaussian noise
Closed form solution in every step
![Page 67: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/67.jpg)
Pose Estimation vs. Pose Tracking
17/02/2015 Computer Vision I: Tracking 67
One Shot Pose Estimation [1] Pose Tracking
[1] Brachmann, E., Krull, A., Michel, F., Shotton, J., Gumhold, S., Rother, C.: Learning 6d object pose estimation using 3d object coordinates, ECCV (2014)
• Estimate 6D Pose from singleRGB-D image
• Use Object Coordinate Regression
• Stream of RDB-D images
• Use information from previous frames:• Realtime
• Increase robustness, accuracy
![Page 68: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/68.jpg)
• Augmented Reality• Alteration• Annotation• Substitution
Robotics Recognition/Tracking
Automatic Grasping
Application Scenarios
17/02/2015 Computer Vision I: Tracking 68
![Page 69: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/69.jpg)
Object Coordinate Regression
17/02/2015 Computer Vision I: Tracking 69
Energy OptimizationCompare observed and rendered images
![Page 70: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/70.jpg)
Energy Formulation
17/02/2015 Computer Vision I: Tracking (Part I) 70
𝐸𝑐 Hc = λdepthEcdepth
Hc + 𝜆𝑐𝑜𝑜𝑟𝑑𝐸𝑐𝑐𝑜𝑜𝑟𝑑 𝐻𝑐 + 𝜆𝑜𝑏𝑗𝐸𝑐
𝑜𝑏𝑗𝐻𝑐
Ecdepth
Hc 𝐸𝑐𝑐𝑜𝑜𝑟𝑑 𝐻𝑐 𝐸𝑐
𝑜𝑏𝑗𝐻𝑐 𝐸𝑐 Hc
• Arbitrary 6DoF pose hypotheses 𝐻𝑐 is scored according to:
Channel
Energy
Comparison of Depth Comparison to Forest Prediction
![Page 71: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/71.jpg)
Efficient Search
17/02/2015 Computer Vision I: Tracking 71
Forest Prediction
ObjectSpace
Input
CameraSpace
Pose Hypothesis
![Page 72: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/72.jpg)
How to Adapt it for Pose Tracking?
17/02/2015 Computer Vision I: Tracking 72
Ingredients:
Particle Filter
Object Coordinate Regression
Efficient Search
Observation Model:
Proposal Distribution:• Find rough estimate using
• Concentrate samples aroundrough estimate
Energy
Motion Model:• Assume continous motion
![Page 73: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/73.jpg)
Filtering with Object Coordinates
17/02/2015 Computer Vision I: Tracking 73
Sampling from Proposal DistributionSampling from Motion Model
![Page 74: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/74.jpg)
Filtering with Object Coordinates
17/02/2015 Computer Vision I: Tracking 74
• Find a rough estimate for current pose Efficient Search
Sampling from Motion Model Sampling from Proposal Distribution
![Page 75: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/75.jpg)
Filtering with Object Coordinates
17/02/2015 Computer Vision I: Tracking 75
Sampling from Proposal DistributionSampling from Motion Model
𝑝 𝑥𝑡+1 𝑥𝑡 𝑞 𝑥𝑡+1 𝑥𝑡
![Page 76: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/76.jpg)
Filtering with Object Coordinates
17/02/2015 Computer Vision I: Tracking 76
• Most samples have a very low weight
Sampling from Proposal DistributionSampling from Motion Model
• Few samples have very low weight
𝑤𝑡𝑖= 𝑝(𝑧𝑡|𝑥𝑡 = 𝑥𝑡
𝑖) 𝑤𝑡𝑖= 𝑝(𝑧𝑡|𝑥𝑡 = 𝑥𝑡
𝑖)𝑝 𝑥𝑡 = 𝑥𝑡
𝑖 𝑥𝑡−1 = 𝑥𝑡−1𝑖
𝑞 𝑥𝑡 = 𝑥𝑡𝑖 𝑥𝑡−1 = 𝑥𝑡−1
𝑖
![Page 77: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/77.jpg)
Filtering with Object Coordinates
17/02/2015 Computer Vision I: Tracking 77
• Most samples have a very low weight
• Few samples have very low weight
Sampling from Proposal DistributionSampling from Motion Model
𝑤𝑡𝑖= 𝑝(𝑧𝑡|𝑥𝑡 = 𝑥𝑡
𝑖) 𝑤𝑡𝑖= 𝑝(𝑧𝑡|𝑥𝑡 = 𝑥𝑡
𝑖)𝑝 𝑥𝑡+1 𝑥𝑡 = 𝑥𝑡
𝑖
𝑞 𝑥𝑡+1 𝑥𝑡 = 𝑥𝑡𝑖
![Page 78: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/78.jpg)
Filtering with Object Coordinates
17/02/2015 Computer Vision I: Tracking 78
Sampling from Proposal DistributionSampling from Motion Model
• More efficient
• Number of Particles can be reduced
• Frame rate can be increased
![Page 79: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/79.jpg)
Evaluation: Choi and Christensen’s Dataset [2]
17/02/2015 Computer Vision I: Tracking 79
• A total of 4 synthetic sequences with 4 objects
• Objects placed in static environment
• Camera moving around object
• Partial occlusion
• Very exact ground truth
[2] Changhyun Choi, Henrik I. Christensen, RGB-D Object Tracking: A Particle Filter Approach on GPU, IROS, 2013
![Page 80: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/80.jpg)
Evaluation: Our Dataset
17/02/2015 Computer Vision I: Tracking 80
• A total of 6 captured sequences of with 3 objects
• Manually annotated ground truth
• Moving objects in front of dynamic background
• Fast erratic movement
• Strong occlusions
![Page 81: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/81.jpg)
Evaluation: Our Dataset
17/02/2015 Computer Vision I: Tracking 81
• Compared to [1] applied to each frame separately
• Lower average error
• Almost no outliers
![Page 82: Computer Vision I - Tracking › ~ds24 › lehre › cv1_ws_2014 › VL13.pdf · Computer Vision I - Tracking Carsten Rother 17/02/2015 Computer Vision I: Tracking 17/02/2015 [slide](https://reader031.vdocuments.site/reader031/viewer/2022041116/5f276a03098afe549d0e2b7d/html5/thumbnails/82.jpg)
Conclusion
17/02/2015 Computer Vision I: Tracking 82
• We have adapted the systemfrom [1] for real time pose tracking
• We have designed a proposal distribution making efficient use of object coordinates
• Our method is robust against:• Quick motion
• Strong occlusion
• Shadows and changing light conditions
• Deformation
[1] Brachmann, E., Krull, A., Michel, F., Shotton, J., Gumhold, S., Rother, C.: Learning 6d Object Pose Estimation Using 3d Object Coordinates, ECCV '14 (2014)