reconstructing relief surfaces george vogiatzis, philip torr, steven seitz and roberto cipolla bmvc...
TRANSCRIPT
![Page 1: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/1.jpg)
Reconstructing Relief Surfaces
George Vogiatzis, Philip Torr, Steven George Vogiatzis, Philip Torr, Steven Seitz and Roberto CipollaSeitz and Roberto Cipolla
BMVC 2004BMVC 2004
![Page 2: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/2.jpg)
Stereo reconstruction problem:
InputInput Set of images of a scene I={ISet of images of a scene I={I11,…,I,…,IKK}}
Camera matrices PCamera matrices P11,…,P,…,PKK
OutputOutput Surface modelSurface model
![Page 3: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/3.jpg)
Shape parametrisation
Disparity-map parametrisationMRF formulation – good optimisation
techniques exist (Graph-cuts, Loopy BP)MRF smoothness is viewpoint dependentDisparity is unique per pixel – only
functions represented
![Page 4: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/4.jpg)
Shape parametrisation
Volumetric parametrisation – e.g. Level-sets, Space carving etc.Able to cope with non-functionsConvergence properties not well
understood, Local minimaMemory intensiveFor Space carving, no simple way to
impose surface smoothness
![Page 5: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/5.jpg)
Solution ?
Cast volumetric methods in MRF frameworkCast volumetric methods in MRF framework Key assumption: Approximate scene geometry Key assumption: Approximate scene geometry
givengiven Benefits:Benefits:
General surfaces can be representedGeneral surfaces can be represented Optimisation is tractable (MRF solvers)Optimisation is tractable (MRF solvers) Occlusions are approximately modelledOcclusions are approximately modelled Smoothness is viewpoint independentSmoothness is viewpoint independent
![Page 6: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/6.jpg)
MRFs
The labelling problem:The labelling problem:
![Page 7: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/7.jpg)
MRFs
A set of random variables hA set of random variables h11,…,h,…,hMM
A binary neighbourhood relation A binary neighbourhood relation NN defined on the defined on the variablesvariables
Each can take a label out of a set HEach can take a label out of a set H11,…,H,…,HLL
CCii(h(hii) (Labelling cost)) (Labelling cost) CCi,ji,j(h(hii,h,hjj) for (i,j)) for (i,j)N N (Compatibility cost)(Compatibility cost)
-log P(h-log P(h11,…,h,…,hMM) = ) = CCii(h(hii) + ) + CCi,ji,j(h(hii,h,hjj) )
![Page 8: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/8.jpg)
MRF inference
Minimise Minimise CCii(h(hii) + ) + CCi,ji,j(h(hii,h,hjj) ) Not in polynomial time in general caseNot in polynomial time in general case Special cases (e.g. no loops or 2 label MRF) Special cases (e.g. no loops or 2 label MRF)
solved exactly solved exactly General cases solved approximately via General cases solved approximately via
Graph-cuts or Loopy Belief Propagation. Graph-cuts or Loopy Belief Propagation. Approx. 10-15mins for MRF with 250,000 Approx. 10-15mins for MRF with 250,000 nodes.nodes.
![Page 9: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/9.jpg)
Relief Surfaces Approximate base surfaceApproximate base surface
Triangulated feature matchesTriangulated feature matches Visual hull from silhouettesVisual hull from silhouettes Initialised by handInitialised by hand
![Page 10: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/10.jpg)
Relief Surfaces labels :
![Page 11: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/11.jpg)
Relief Surfaces labelling cost :
Low costHigh cost
Xi
ni
Xi+hini
Ci(hi)=photoconsistency(Xi+hini)
![Page 12: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/12.jpg)
Relief Surfaces Compatibility cost :
Low cost
Xi
ni
nj Xj
Xi+hini
Xj+hjnj
![Page 13: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/13.jpg)
Relief Surfaces Neighbour cost :
Ci,j(hi, hj)= ||(Xi+hini)-(Xj+hjnj)||
Xi
ni
Xi+hini
Xj+hjnj
High cost
![Page 14: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/14.jpg)
Relief Surfaces
Base surface is the Base surface is the occluding volumeoccluding volume If base surface ‘contains’ true surface (e.g. visual If base surface ‘contains’ true surface (e.g. visual
hull) thenhull) then Points on the base surface Points on the base surface XXii are not visible by are not visible by
cameras they shouldn’t be [Kutulakos, Seitz cameras they shouldn’t be [Kutulakos, Seitz 2000]2000]
Approximation:Approximation: Visibility is propagated from Visibility is propagated from XXii to to XXii+h+hiinnii
![Page 15: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/15.jpg)
Loopy Belief Propagation
Iterative message passing algorithmIterative message passing algorithm mm(t)(t)
i,ji,j (h (hjj) is the message passed from i to j at time step t) is the message passed from i to j at time step t
It is a L-dimensional vectorIt is a L-dimensional vector Represents what node i ‘believes’ about the true state of Represents what node i ‘believes’ about the true state of
node j.node j.
min min CCii(h(hii) + ) + CCi,ji,j(h(hii,h,hjj))
i jmi,j
![Page 16: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/16.jpg)
Loopy Belief Propagation
Message passing rule:Message passing rule:
After convergence, optimal state is given byAfter convergence, optimal state is given by
mm(t+1)(t+1)i,ji,j (h (hjj)= min)= min{{ C Cijij(h(hii,h,hjj) +C) +Cii(h(hii) +) +mm(t)(t)
k,ik,i (h (hii))}}k N(i)hi
hhii**= min= min{{CCii(h(hii) +) +mm(())
k,ik,i (h (hii))}}k N(i)hi
i jmi,j
![Page 17: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/17.jpg)
Loopy Belief Propagation
O(LO(L22) to compute a message (L is number of ) to compute a message (L is number of allowable heights)allowable heights)
Message passing schedule can be asynchronous Message passing schedule can be asynchronous which can accelerate convergence [Tappen & which can accelerate convergence [Tappen & Freeman ICCV 03]Freeman ICCV 03]
![Page 18: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/18.jpg)
Iterative Scheme BP is memory intensive. BP is memory intensive. Can consider few possible labels at a timeCan consider few possible labels at a time After convergence we ‘zoom in’ to heights close to the optimalAfter convergence we ‘zoom in’ to heights close to the optimal
![Page 19: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/19.jpg)
Evaluation
Artificial deformed sphereArtificial deformed sphere Textured with random paternTextured with random patern 20 images20 images 40,000 sample points on sphere base surface40,000 sample points on sphere base surface
True surface Texture-mapped Reconstruction
![Page 20: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/20.jpg)
Evaluation
Benchmark: 2-view, disparity based Benchmark: 2-view, disparity based Loopy Loopy Belief Propagation Belief Propagation [Sun et al ECCV02][Sun et al ECCV02]
BP run on 10 pairs of nearby viewsBP run on 10 pairs of nearby views Compare Disparity Maps given by Compare Disparity Maps given by
2-view BP2-view BP Relief surfacesRelief surfaces Ground truthGround truth
![Page 21: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/21.jpg)
Evaluation2-view BP Relief surface Ground truth
2-view BP2-view BP Relief surf.Relief surf.
MSEMSE 1.466 pixels1.466 pixels 0.499 pixels0.499 pixels
% of correct disparities% of correct disparities 75.9%75.9% 79.2%79.2%
![Page 22: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/22.jpg)
Results
SarcophagusSarcophagus
![Page 23: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/23.jpg)
Results
SarcophagusSarcophagus
![Page 24: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/24.jpg)
Results
SarcophagusSarcophagus
![Page 25: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/25.jpg)
Results
Building facadeBuilding facade
![Page 26: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/26.jpg)
Results
Building facadeBuilding facade
![Page 27: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/27.jpg)
Results
Base surface Relief surfaceRelief surfacewith texture
Stone carvingStone carving
![Page 28: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/28.jpg)
Summary
MRF methods can be extended in the MRF methods can be extended in the volumetric domainvolumetric domain
AdvantagesAdvantages General surfaces can be representedGeneral surfaces can be represented Optimisation is tractable (MRF solvers)Optimisation is tractable (MRF solvers) Smoothness is viewpoint independentSmoothness is viewpoint independent
![Page 29: Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004](https://reader036.vdocuments.site/reader036/viewer/2022062500/56649e035503460f94aeded0/html5/thumbnails/29.jpg)
Future work
Photoconsistency beyond Lambertian Photoconsistency beyond Lambertian surface models. (Optimise both height and surface models. (Optimise both height and surface normal fields)surface normal fields)
Change in topologyChange in topology In cases where CIn cases where Cmnmn(h(hmm,h,hnn)=|| h)=|| hmm-h-hnn|||| or || hor || hmm--
hhnn||||2 2 we can compute messages in O(L) time we can compute messages in O(L) time instead of O(Linstead of O(L22) (Felzenszwalb & ) (Felzenszwalb & Huttenlocher CVPR 04).Huttenlocher CVPR 04).