Upload File

semantic parsing for priming object detection in rgb-d scenes

  • Home
  • Documents
  • Semantic Parsing for Priming Object Detection in RGB-D Scenes
47
3rd Workshop On Semantic Perception, Mapping and Exploration (SPME) Karlsruhe, Germany ,2013 Semantic Parsing for Priming Object Detection in RGB-D Scenes Cesar Cadena and Jana Kosecka

Upload: ninon

Post on 22-Feb-2016

79 views

Category:

Documents


0 download

Report

Tags:

DESCRIPTION

Semantic Parsing for Priming Object Detection in RGB-D Scenes. Cesar Cadena and Jana Kosecka. 3rd Workshop On Semantic Perception, Mapping and Exploration (SPME) Karlsruhe, Germany ,2013. Motivation. Long-term robotic operation - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Semantic Parsing for Priming Object Detection in RGB-D Scenes

3rd Workshop On Semantic Perception, Mapping and Exploration (SPME)Karlsruhe, Germany ,2013

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Cesar Cadena and Jana Kosecka

Page 2: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Motivation

5/5/2013

Long-term robotic operation

The semantic information about the surrounding environment is important for high level robotic tasks.

It is difficult to know a priori all the possible instances or classes of objects that the robot will find in a real operation.

Even if we know a lot of them, it is unreasonable and expensive, run all specific object detectors at the same time.

Page 3: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Motivation

5/5/2013

Long-term robotic operation

The semantic information about the surrounding environment is important for high level robotic tasks.

It is difficult to know a priori all the possible instances or classes of objects that the robot will find in a real operation.

Even if we know a lot of them, it is unreasonable and expensive, run all specific object detectors at the same time.

Page 4: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Motivation

5/5/2013

Long-term robotic operation

The semantic information about the surrounding environment is important for high level robotic tasks.

It is difficult to know a priori all the possible instances or classes of objects that the robot will find in a real operation.

Even if we know a lot of them, it is unreasonable and expensive, run all specific object detectors at the same time.

Page 5: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Motivation

5/5/2013

Long-term robotic operation

The semantic information about the surrounding environment is important for high level robotic tasks.

It is difficult to know a priori all the possible instances or classes of objects that the robot will find in a real operation.

Even if we know a lot of them, it is unreasonable and expensive, run all specific object detectors at the same time.

Page 6: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

However: There are things we can assume to be present

(almost) always Generic “detachable” objects also share some

characteristics

Urban: Ground Buildings Sky ObjectsIndoors: Ground Walls Ceiling Objects

Today: Ground – Structure – Furniture – Props

Efficiently to segment RGB+3D scenes into these general classes to be used as a prior for specific task detectors

Motivation

5/5/2013

Page 7: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

However: There are things we can assume to be present

(almost) always Generic “detachable” objects also share some

characteristics

Urban: Ground Buildings Sky ObjectsIndoors: Ground Walls Ceiling Objects

Today: Ground – Structure – Furniture – Props

Efficiently to segment RGB+3D scenes into these general classes to be used as a prior for specific task detectors

Motivation

5/5/2013

Page 8: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

However: There are things we can assume to be present

(almost) always Generic “detachable” objects also share some

characteristics

Urban: Ground Buildings Sky ObjectsIndoors: Ground Walls Ceiling Objects

Today: Ground – Structure – Furniture – Props

Efficiently to segment RGB+3D scenes into these general classes to be used as a prior for specific task detectors

Motivation

5/5/2013

Page 9: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

However: There are things we can assume to be present

(almost) always Generic “detachable” objects also share some

characteristics

Urban: Ground Buildings Sky ObjectsIndoors: Ground Walls Ceiling Objects

Today: Ground – Structure – Furniture – Props

Efficiently to segment RGB+3D scenes into these general classes to be used as a prior for specific task detectors

Motivation

5/5/2013

Page 10: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

However: There are things we can assume to be present

(almost) always Generic “detachable” objects also share some

characteristics

Urban: Ground Buildings Sky ObjectsIndoors: Ground Walls Ceiling Objects

Today: Ground – Structure – Furniture – Props

Efficiently to segment RGB+3D scenes into these general classes to be used as a prior for specific task detectors

Our Problem

5/5/2013

Page 11: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

However: There are things we can assume to be present

(almost) always Generic “detachable” objects also share some

characteristics

Urban: Ground Buildings Sky ObjectsIndoors: Ground Walls Ceiling Objects

Today: Ground – Structure – Furniture – Props

Efficiently to segment RGB+3D scenes into these general classes to be used as a prior for specific task detectors

Our Problem

5/5/2013

Page 12: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

NYU Depth v2

5/5/2013

1449 labeled frames. 26 scenes classes. Labeling spans over 894 different classes.

N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, Indoor segmentation and support inference from RGBD images, in ECCV, 2012.

Thanks to N. Silberman for proving the mapping 894 to 4 classes.

Page 13: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

The System

5/5/2013

Semantic Segmentation

MAPMarginals

Page 14: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Different approaches

5/5/2013

Semantic Segmentation

MAPMarginals

N. Silberman et al. ECCV 2012 C. Couprie et al. CoRR 2013 X. Ren et al. CVPR 2012 D. Munoz et al. ECCV 2010 I. Endres and D. Hoeim, ECCV

2010

They have at least one:

Expensive over-segmentation

Expensive features Expensive Inference

Page 15: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Our approach

5/5/2013

MAPMarginals

Semantic Segmentation

Conditional Random Fields

Potentials

Graph Structure Inferenc

ePreprocessing

Page 16: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Outline

5/5/2013

MAPMarginals

Conditional Random Fields

Potentials

Graph Structure Inferenc

ePreprocessing (1)

(2)(3)

(5)Results(6)Conclusio

ns

(4)

Page 17: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Preprocessing: Over-segmentation

5/5/2013

SLIC superpixels

R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk,SLIC superpixels compared to state-of-the-art superpixel methods,PAMI, 2012.

Page 18: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Graph Structure

5/5/2013

Classical choice on images

Page 19: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Graph Structure: Our choice

5/5/2013

Minimum Spanning Tree

Over 3D

Page 20: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Graph Structure: Our choice

5/5/2013

Minimum Spanning Tree

Over 3D

Page 21: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Potentials: Pairwise CRFs

5/5/2013

Page 22: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Potentials: Pairwise CRFs

5/5/2013

Page 23: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Potentials: Pairwise CRFs

5/5/2013

Page 24: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Potentials: unary

5/5/2013

frequency of label j in a k-NN queryfrequency of label j the database

J. Tighe and S. Lazebnik, Superparsing: Scalable nonparametric image parsing with superpixels,ECCV 2010.

The database is a kd-tree of features from training data

Page 25: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Features 12D

5/5/2013

From Image: mean of Lab color space

3D vertical pixel location

1D entropy from vanishing points

1D

From 3D height and depth

2D mean and std of differences on depth

2D local planarity

1D neighboring planarity

1D vertical orientation

1D

Page 26: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Features

5/5/2013

From Image: entropy from vanishing points

Page 27: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Features

5/5/2013

From 3D mean and std of differences on depth

Page 28: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Features

5/5/2013

From 3D mean and std of differences on depth

Page 29: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Features

5/5/2013

From 3D mean and std of differences on depth

local planarity neighboring planarity vertical orientation

Page 30: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Potentials: pairwise

5/5/2013

Lab color

Page 31: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Inference

5/5/2013

We use belief propagation:

Exact results in MAP/marginals

Efficient computation, in

Thanks to our graph structure choice!

Page 32: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Results: NYU-D v2 Dataset

5/5/2013

GT MAP

Page 33: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Results: NYU-D v2 Dataset

5/5/2013

Confusion matrix:

Comparisons:

Page 34: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Results: NYU-D v2 Dataset

5/5/2013

Confusion matrix:

Comparisons:

Page 35: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Results: NYU-D v2 Dataset

5/5/2013

GT MAP

Some failures:

Page 36: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Results: NYU-D v2 Dataset

5/5/2013

Page 37: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Marginal probabilities

5/5/2013

Provide very useful information for specific tasks, e.g. : Specific object detection Support inference

P(Ground) P(Structure) P(Furniture) P(Props)

Page 38: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Conclusions

5/5/2013

We have presented a computational efficient approach for semantic segmentation of priming objects in indoors.

Our approach effectively uses 3D and Images cues.

Depth discontinuities are evidence for occlusions

The MST over 3D keeps intra-class components coherently connected.

Page 39: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Discussion

5/5/2013

Features:

Local classifier:

Graph structure

Bunch of engineered features (>1000D)

Learned features(>1000D)

Select meaningful features(12D)

Logistic Regression Neural Networks k-NN

Dense ConnectionsImage

None MST over 3D

Silberman et al. 2012 Couprie et al. 2013

Ours.

Page 40: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Thanks!!

5/5/2013

Cesar Cadena [email protected] Kosecka [email protected]

Funded by the US Army Research Office Grant W911NF-1110476.

Page 41: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Working on:

5/5/2013

People detection by Shenghui Zhou

Page 42: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Multi-view and video:

5/5/2013

Page 43: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Multi-view and video:

5/5/2013

Page 44: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Multi-view and video:

5/5/2013

Page 45: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Multi-view and video:

5/5/2013

Page 46: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Multi-view and video:

5/5/2013

Page 47: Semantic Parsing for Priming Object Detection in RGB-D Scenes

Semantic Parsing for Priming Object Detection in RGB-D Scenes

Multi-view and video:

5/5/2013


Road priming

Statistical Dependency Parsing - Uppsala Universitynivre/docs/StatState.pdf · I Methods for statistical dependency parsing I Chart parsing techniques I Parsing as constraint satisfaction

Parsing: Top-Down vs. Bottom-Up Parsing Algorithms Partial Parsing

Malleable 2.5D Convolution: Learning Receptive Fields ...€¦ · Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsing Yajie Xing1[0000

Bare-Bones Dependency Parsing - Uppsala Universitystp.lingfil.uu.se/~nivre/docs/BareBones.pdf · I Parsing methods for bare-bones dependency parsing I Chart parsing ... Eisner 2000]:

Weighted Parsing, Probabilistic Parsing

3rd Workshop On Semantic Perception, Mapping and Exploration (SPME) Karlsruhe, Germany,2013 Semantic Parsing for Priming Object Detection in RGB-D Scenes

Holistic3D Scene Parsing and Reconstruction from a Single ...openaccess.thecvf.com/content_ECCV_2018/papers/... · Holistic3D Scene Parsing and Reconstruction from a Single RGB Image

DMX RGB, RGB-WHITE, RGB-AMBER & TUNABLE WHITE • 0-10V …€¦ · • DMX RGB, RGB-WHITE, RGB-AMBER & TUNABLE WHITE • 0-10V WARM DIM DIMENSIONAL INFORMATION C3A-MTGBRKT-SHIELD

Priming Kanban

Dependency Parsing (3) - University Of Maryland · Dependency Parsing: what you should know •Transition-based dependency parsing •Shift-reduce parsing •Transition systems: arc

Basic Parsing Algorithms: Earley Parser and Left Corner Parsing Alexandr Chernov Recent Advances in Parsing Technology

RGB: 227/227/227 RGB: 196/196/196 RGB: 158/158/158 RGB: 115/115/115 RGB: 64/64/64 RGB: 0/0/0 RGB: 249/214/192 RGB: 243/172/128 RGB: 234/131/64 RGB: 229/90/0

Syntactic Priming

Top Down Parsing, Predictive Parsing

RGB Twitter: #AFF2018 #10YearsOn #ChangeFinance RGB · RGB 70-190-165 RGB 245-125 -43 RGB 237-185-29 RGB 47-127-111 RGB 24-64-55 RGB 127 65 22 64 33 11 RGB 127-100-16 RGB 64-50-8

Bare-Bones Dependency Parsing - Uppsala Universitynivre/docs/BareBones.pdf · I Parsing methods for bare-bones dependency parsing I Chart parsing techniques ... [Kuhlmann and Satta

Priming Process

October 2008csa3180: Setence Parsing Algorithms 1 1 CSA350: NLP Algorithms Sentence Parsing I The Parsing Problem Parsing as Search Top Down/Bottom Up

Top-Down Parsing - recursive descent - predictive parsing

RGB controller 4 RGB controller 11 Controlador RGB 32 · Contrôleur RGB 18 RGB controller 4 RGB controller 11. 1) On/off switch 2) Mode switch. ... Modo "RGB": ajuste por separado

Parsing. 2301373Chapter 4 Parsing2 Outline Top-down v.s. Bottom-up Top-down parsing Recursive-descent parsing LL(1) parsing LL(1) parsing algorithm First

IV Priming

Tiras RGB de LED‘s SMD 5050 - Amazon Web Services · 2015-08-10 · RGB-150 @ 36w / RGB-300 @ 72W RGB-150 @ 7.2W / RGB-300 @ 14.4W RGB-150 @1800lm* /RGB-300 @2160 lm* Universales,

TECHNICAL DATA SELF-PRIMING AND MULTISTAGE CENTRIFUGAL PUMPS · SELF-PRIMING AND MULTISTAGE CENTRIFUGAL PUMPS JETCOM SELF-PRIMING CENTRIFUGAL ELECTRIC PUMPS APPLICATIONS Self priming

Syntactic Analysis Operator-Precedence Parsing Recursive-Descent Parsing

Priming Unconscious Racial Stereotypes About Adolescent ...stopyouthviolence.ucr.edu/pubs_by_topic/4.Priming unconscious... · Priming Unconscious Racial Stereotypes About Adolescent

Dependency Parsing - Oregon State Universityclasses.engr.oregonstate.edu/.../notes/dependency-parsing-lecture.pdf · Outline Introduction Dependency Parsing Formal definition Parsing

Parsing Context-Free Grammars, Parsing, Syntax Trees

Chart Parsing and Probabilistic Parsing

PARSING WITH CONTEXT-FREE GRAMMARS cc437. PARSING Parsing is the process of recognizing and assigning STRUCTURE Parsing a string with a CFG: – Finding

3D Semantic Parsing of Large-Scale Indoor Spacessvl.stanford.edu/assets/papers/3D_Semantic_Parsing.pdfmethods that use both RGB and depth [27, 32, 8]. How-ever, the 3D sensing field

Parsing Indoor Scenes Using RGB-D Imagery · the parsing problem. The proposed approach is also similar to the parsing scheme described by Flint et al.[3, 2] who also make use of

Chart Parsing and Probabilistic Parsing - SourceForgenltk.sourceforge.net/doc/en/advanced-parsing.pdfChart Parsing and Probabilistic Parsing 9.1 Introduction ... Furthermore, it is

– 1 – CSCE 531 Spring 2006 Lecture 8 Bottom Up Parsing Topics Overview Bottom-Up Parsing Handles Shift-reduce parsing Operator precedence parsing Readings: