lecture 18: depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • ps9...
TRANSCRIPT
![Page 1: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/1.jpg)
1
Lecture 18: Depth estimation
![Page 2: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/2.jpg)
• PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA) • Final presentation will take place over video chat. - We’ll send a sign-up sheet next week
2
Announcements
![Page 3: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/3.jpg)
Today
• Stereo matching • Probabilistic graphical models • Belief propagation • Learning-based depth estimation
3
![Page 4: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/4.jpg)
Basic stereo algorithm
Foreachepipolarline Foreachpixelintheleftimage
• comparewitheverypixelonsameepipolarlineinrightimage
• pickpixelwithminimummatchcost
Improvement:matchwindowsSource: N. Snavely4
![Page 5: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/5.jpg)
Stereo matching based on SSD
SSD
dmin d
Best matching disparity
Source: N. Snavely5
![Page 6: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/6.jpg)
Window size
W = 3 W = 20
Source: N. Snavely6
![Page 7: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/7.jpg)
Stereo as energy minimization
• Whatdefinesagoodstereocorrespondence?1. Matchquality• Wanteachpixeltofindagoodmatchintheotherimage
2. Smoothness• Iftwopixelsareadjacent,theyshould(usually)moveaboutthe
sameamountSource: N. Snavely7
![Page 8: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/8.jpg)
Stereoasenergyminimization
• Finddisparitymapdthatminimizesanenergyfunction
• Simplepixel/windowmatching
SquareddistancebetweenwindowsI(x,y)andJ(x+d(x,y),y)=
Source: N. Snavely8
![Page 9: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/9.jpg)
Stereoasenergyminimization
I(x,y) J(x,y)
y=141
C(x,y,d);thedisparityspaceimage(DSI)x
d
Source: N. Snavely9
![Page 10: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/10.jpg)
Stereoasenergyminimization
y=141
x
d
Simplepixel/windowmatching:choosetheminimumofeachcolumnintheDSIindependently:
Source: N. Snavely10
![Page 11: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/11.jpg)
Greedyselectionofbestmatch
Source: N. Snavely11
![Page 12: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/12.jpg)
Stereo as energy minimization• Betterobjectivefunction
{ {matchcost smoothnesscost
Wanteachpixeltofindagoodmatchintheotherimage
Adjacentpixelsshould(usually)moveaboutthesameamount
Source: N. Snavely12
![Page 13: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/13.jpg)
Stereo as energy minimization
matchcost:
smoothnesscost:
4-connectedneighborhood
8-connectedneighborhood
:setofneighboringpixels
Source: N. Snavely13
![Page 14: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/14.jpg)
Smoothness cost
“Pottsmodel”
L1distance
HowdowechooseV?
Source: N. Snavely14
![Page 15: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/15.jpg)
Probabilistic interpretation
exp(E(d)) = exp(Ed(d) + �Es(d))Exponentiate:
exp(E(d))
Z=
1
Zexp(Ed(d) + �Es(d))
Z =
X
d0
expE(d0)where
Normalize: (make it sum to 1)
Rewrite: P (d | I)
Example adapted from Freeman, Torralba, Isola15
![Page 16: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/16.jpg)
Probabilistic interpretation
P (d | I)
“Local evidence” “Pairwise compatibility”How good are the matches? Is the depth smooth?
Example adapted from Freeman, Torralba, Isola16
![Page 17: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/17.jpg)
Probabilistic interpretation
P (d | I)
Local evidence: Pairwise compatibility:
Example adapted from Freeman, Torralba, Isola17
![Page 18: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/18.jpg)
Probabilistic graphical models
P (d | I)
Graph structure: • Open circles for latent variables xi
- di in our problem • Filled circle for observations yi
- Pixels in our problem • Edges between interacting variables
- In general, graph cliques for 3+ variable interactions
Example adapted from Freeman, Torralba, Isola18
![Page 19: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/19.jpg)
Probabilistic graphical models
P (d | I)
Why formulate it this way? • Exploit sparse graph structure for fast inference,
usually using dynamic programming • Can use probabilistic inference methods • Provides framework for learning parameters
19
![Page 20: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/20.jpg)
Directed graphical model Also know as Bayesian network (Not covered in this course)
Probabilistic graphical models
Undirected graphical model. Also known as Markov Random Field (MRF).
20
![Page 21: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/21.jpg)
Marginalization
What’s the marginal distribution for x1? i.e. what’s the probability of x1 being in a particular state?
• But this is expensive: O(|L|^N) • Exploit graph structure! 21
![Page 22: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/22.jpg)
Marginalization
22
![Page 23: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/23.jpg)
Message passing
Message that node x3 sends to node x2
Message that x2 sends to x1
Can think of “local evidence” message passing
23
![Page 24: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/24.jpg)
• Message mij is the sum over all states of all nodes in the subtree leaving node i at node j
• It summarizes what this node “believes”. • E.g. if you have label x2, what’s the probability of my subgraph?
• Shared computation! E.g. could reuse m32 to help estimate p(x2 | y).
Message passing
24
![Page 25: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/25.jpg)
Belief propagation• Estimate all marginals p(xi | y) at once!
[Pearl 1982] • Given a tree-structured graph, send
messages in topological order
Sending message from j to i:1. Multiply all incoming messages
(except for the one from i) 2. Multiply the pairwise compatibility 3. Marginalize over xj
25
![Page 26: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/26.jpg)
General graphs• Vision problems often are often on grid graphs • Pretend the graph is tree-structured and do belief
propagation iteratively! • Can also have consistency with N > 2 variables
- But complexity is exponential in N!
Loopy belief propagation:1. Initialize all messages to 1 2. Walk through the edges in an
arbitrary order (e.g. random) 3. Apply the messages updates
26
![Page 27: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/27.jpg)
Finding best labels
Marginal: “Max marginal” instead:
P (x1|~y) = max
x2,x3b(x1 | ~y) =
27
argmax
max
x1,x2,x3
Often want to find the labels that jointly maximize probability:
max
xj
This is called maximum a posteriori estimation (MAP estimation).
![Page 28: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/28.jpg)
Application to stereo
[Felzenzwalb & Huttenlocher, “Efficient Belief Propagation for Early Vision”, 2006]
28
![Page 29: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/29.jpg)
Deep learning + MRF refinement
[Zbontar & LeCun, 2015]
Left
Right
Positive NegativeQuery patch
CNN-based matching + MRF refinement
29
![Page 30: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/30.jpg)
Learning to estimate depth without ground truth
[Mildenhall*, Srinivasan*, Tanick*, et al., Neural radiance fields, 2020]
3D scene Viewpoints
Learn “volume”: color + occupancy
30
![Page 31: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/31.jpg)
Learning to estimate depth without ground truth
A good volume should reconstruct the input views
[Mildenhall*, Srinivasan*, Tanick*, et al. 2020]31
![Page 32: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/32.jpg)
Learning to estimate depth without ground truth
[Mildenhall*, Srinivasan*, Tanick*, et al. 2020]32
![Page 33: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/33.jpg)
Learning to estimate depth without ground truth
[Mildenhall*, Srinivasan*, Tanick*, et al. 2020]
Inserting virtual objects View synthesis33
![Page 34: Lecture 18: Depth estimation - web.eecs.umich.eduahowens/eecs504/w20/slides/lec19-de… · • PS9 out tonight: panorama stitching • New grading policies from UMich (details TBA)](https://reader036.vdocuments.site/reader036/viewer/2022081402/5f08e5117e708231d4243da3/html5/thumbnails/34.jpg)
Next class: motion
34