panocontext: a whole-room 3d context model for panoramic ... · panocontext: a whole-room 3d...

PanoContext: A Whole-room 3D Context Model

for Panoramic Scene Understanding

Yinda Zhang Shuran Song Ping Tan† Jianxiong Xiao

Princeton University † Simon Fraser University

Alicia Clark PanoContext October 17, 2014 1 / 11

Importance of Context in Image Processing

Alicia Clark PanoContext October 17, 2014 2 / 11

Importance of Context in Image Processing

Alicia Clark PanoContext October 17, 2014 2 / 11

Importance of Context in Image Processing

Alicia Clark PanoContext October 17, 2014 2 / 11

Importance of Context in Image Processing

Alicia Clark PanoContext October 17, 2014 2 / 11

Importance of Context in Image Processing

Context ModelsProposed in the past, but they do not work as well as expected

WHY?

Alicia Clark PanoContext October 17, 2014 2 / 11

Field of View (FOV)

Small FOV =⇒ Hinders the contextual information

Alicia Clark PanoContext October 17, 2014 3 / 11

Field of View (FOV)

Alicia Clark PanoContext October 17, 2014 3 / 11

Field of View (FOV)

Alicia Clark PanoContext October 17, 2014 3 / 11

Panorama - 360◦ Horizontal and 180◦ Vertical

Easily obtained by smartphones, special lenses, or image stitching

All objects are usually visible despite occlusion

Enables the detection of the room layout and of the contextualinformation

Panorama =⇒ Object Detection =⇒ Whole-room 3D ContextModel

Alicia Clark PanoContext October 17, 2014 4 / 11

Panorama - 360◦ Horizontal and 180◦ Vertical

Easily obtained by smartphones, special lenses, or image stitchingAll objects are usually visible despite occlusion

Enables the detection of the room layout and of the contextualinformation

Panorama =⇒ Object Detection =⇒ Whole-room 3D ContextModel

1Manhattan world assumption: assumes the scene consists of 3D cuboidsaligned with the three principle directions

2Assume no floating objectsAlicia Clark PanoContext October 17, 2014 4 / 11

PanoContext: A Whole-Room 3D Context Model

Algorithm1 Generate a set of hypotheses for room layout and objects

2 Rank these whole-room hypotheses holistically to determine thebest hypothesis

Alicia Clark PanoContext October 17, 2014 5 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Figure : Hough transform for vanishing point detection

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Figure : Comparison of OM and GC. OM works better on the top half ofthe image, while GC provides better normal estimation at the bottom

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Figure : Two ways to generate object hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

1Semantic LabelAlicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

1Pairwise ConstraintAlicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

1. Generate a set of whole-room hypotheses

Alicia Clark PanoContext October 17, 2014 6 / 11

PanoContext: A Whole-Room 3D Context Model

2. Determine the best whole-room hypothesesTrain a linear SVM model to rank the whole-room hypothesesand choose the best hypothesis

Want the matching cost (difference between whole-roomhypothesis and its ground truth) to be as low as possible

Alicia Clark PanoContext October 17, 2014 7 / 11

Summary: How does context help?

Helps to determine the size of objects

Helps to determine the correct number of objects

Helps the determine the relative position of objects

Alicia Clark PanoContext October 17, 2014 8 / 11

Summary: How does context help?

Helps to determine the size of objects

Helps to determine the correct number of objects

Helps the determine the relative position of objects

Alicia Clark PanoContext October 17, 2014 8 / 11

Summary: How does context help?

Helps to determine the size of objects

Helps to determine the correct number of objects

Helps the determine the relative position of objects

Alicia Clark PanoContext October 17, 2014 8 / 11

Summary: How does context help?

1DPM: Deformable Part Model2Felzenszwalb et. al: Discriminative training with partially labeled data

Alicia Clark PanoContext October 17, 2014 9 / 11

Contributions

Context model is fully in 3D

First annotated panorama dataset

Alicia Clark PanoContext October 17, 2014 10 / 11

Questions?

Alicia Clark PanoContext October 17, 2014 11 / 11