Object Stereo- Joint Stereo Matching and Object Segmentation

Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on

Michael Bleyer Vienna University of Technology, Austria

Carsten Rother Microsoft Research Cambridge, UK

Pushmeet Kohli Microsoft Research Cambridge, UK

Daniel Scharstein Middlebury College, USA

Sudipta Sinha Microsoft Research Redmond, USA1

Outline

• Introduction• Proposed Model• Energy Minimization• Result• Conclusion

2

Introduction

• A 3D scene is represented as a collection of visually distinct and spatially coherent objects.

• Each object is characterized by three different aspects: • color model• 3D plane• 3D connectivity

3

Introduction

• The proposed method employs object-level color models as a soft constraint to aid depth estimation.

• The proposed method can recover the depth of regions that are fully occluded in one input view.

4

Introduction

• The proposed method models a 3D scene as a collection of 3D objects, assume that1. each object is compact.

2. each object is connected.

3. all visible parts of an object share a similar appearance.

4. scene interpretations with a few large objects.

5

Introduction

• Compactness• objects are coherent.• depth variations within an object are smooth.• objects have a bias towards being planar in 3D.

6

Introduction

• 3D Connectivity• disconnected 2D regions and separated by smaller

depth.

7

Introduction

• Similar Appearance• use color as the only appearance cue.• each object in a scene has a compact distribution

of colors.

• Scene Interpretation• with few objects.• prevent single pixels from being explained as

individual objects.8

Introduction

• Color models introduce a color segmentation into the stereo matching process.• assign untextured regions to the same object.• extend disparities into untextured regions.• capture disparity discontinuities more precisely.

• Assign disparities to small disconnected background regions in complex occlusions.

9

Outline

• Introduction• Proposed Model• Energy Minimization• Result• Conclusion

10

Proposed Model

• Scene Representation, assume that• disparity map is a collection of 3D planes (depth

planes).• estimate object’s depth by a 3D plane (object

plane).• compute a parallax value obtained by subtracting

p’s disparity at each pixel p within an object op.

11

Parallax Model

• Enforce parallax values have a compact distribution within object op.

• The parallax model provides the probability of the occurrence of a specific parallax in object op.

• The proposed model avoid parallaxes that have low probabilities.

12

• An object o ∈ O contains the following parameters: 1. a color model

2. a parallax model

3. an object plane

• F : I → F that assigns each pixel to a depth plane.• .

• O : I → O that assigns each pixel to an object.

Energy Function

13

Energy Function

• Energy function evaluates the quality of F and O.

• Minimize the energy to obtain a “good” approximation to the Maximum a Posteriori (MAP) solution of the model.

• .

14

Photo Consistency Term Epc

• .

• Measures the pixel dissimilarity of corresponding points and accounts for occlusion handling.

• Ensures that corresponding pixels are assigned to the same depth plane and object.

15

Photo Consistency Term Epc

• .

16

Object-Coherency Term Eoc

• .

• Encourages neighboring pixels in the image to take the same object label.

• . [19]

17

[19] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 23:309–314, 2004.

Depth Plane-Coherency Term Edc

• .

• Depth plane assignments within an object shall be spatially coherent.

• .

18

Object-Color Term Ecol

• .

• Each object contains a color model implemented as a Gaussian Mixture Model (GMM).

• The GMM gives the probability that a pixel lies inside the object according to its color value.

19

Object-Color Term Ecol

• . [19]

20

[19] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 23:309–314, 2004.

Object-Parallax Term Epar

• .

• The disparity at pixel p according to op’s object plane by .

• The parallax is then computed as .

21

Object-Parallax Term Epar

• Distribution of the parallax within same object is likely to be compact.

• .

22

Object-MDL Term Emdl

• .

• The term puts a penalty on the occurrence of an object [4].

• .

23

[4] M. Bleyer, C. Rother, and P. Kohli. Surface stereo with soft segmentation. In CVPR, 2010.

3D Connectivity Econ

• .

• An object is considered connected • a path connects all pixels with the same object label.

• The path are either 1. pixels belong to the same object.

2. pixels belong to different objects.

24

3D Connectivity Econ

• .

• .

25

Outline

• Introduction• Proposed Model• Energy Minimization• Proposal Generator

• Result• Conclusion

26

Energy Minimization

• Proposed model is formulated as an energy function that is optimized via fusion moves [16].

• In the fusion move, a new solution generated by “selecting” • depth planes and objects from S• others from S’

27

[16] V. Lempitsky, C. Rother, and A. Blake. Logcut - efficient graph cut optimization for Markov Random Fields. In ICCV, 2007.

Energy Minimization

• Start with an initial solution S that consists of a disparity map F and an object map O.

• Obtain a proposal S’ from a proposal generator.

• S and S’ are fused to produce a new solution S*.• S := S*

28

Proposal Generator S’

• Initial Proposals : • initialize the disparity map.• color segmentation by mean-shift.• derive F, O.• estimate parameters.• derive a large variety of initial proposals

(approximately 30 ).

29

Proposal Generator S’

• Refit Proposals :• compute a new color model, object plane, parallax

model.

• 〈 F’, O’〉 is derived by refitting the object parameters of the current solution〈 F, O〉 .

30

Proposal Generator S’

• Expansion Proposals : • select one depth plane f present in F and one object

o present in O.

•〈 F’, O’〉 is derived by setting all pixels of F’ to f and all pixels of O’ to o.

31

Optimal Fusion

• Use quadratic pseudo-boolean optimization function (QPBO-F) [11] to the fusion move problem.

• Reduces the problem with multi-valued variables to a sequence of minimization sub-problems with binary variables.

32

[11] V. Kolmogorov and C. Rother. Minimizing non-submodular functions with graph cuts - a review. PAMI, 29(7):1274–1279, 2007.

Outline

• Introduction• Proposed Model• Energy Minimization• Result• Conclusion

33

Result

34

Result

35

Result

36

Outline

• Introduction• Proposed Model• Energy Minimization• Result• Conclusion

37

Conclusion

• The object level enables our algorithm to utilize color segmentation as a soft constraint and to handle difficult occlusion cases.

• A 3D connectivity constraint that enforces consistency of object assignments with stereo geometry.

• Currently, our algorithm is slow, i.e., it takes approximately 20 minutes to obtain results on images.

38