Deep Feature Learning for Contour Detection

Wei Shen (沈为) http://wei-shen.weebly.com/

Shanghai University

Shanghai University 1

Vision And Learning SEminar (VALSE)

Outline 2

Contour Detection Overview

Milestones

Our Work

Discussions

Introduction of Contour Detection 3

Reduce dimensionality of data

Preserve content information

Useful in applications such as

Object Detection Image Segmentation Sketch-based Image Search

Introduction of Contour Detection 4

Edge Detection

the characteristic changes in brightness, color, and texture

Contour Detection

The changes in pixel ownership from one object or surface to another

easy difficult

Challenge 5

To differentiate contour and non-contour is difficult, even for human beings!

Challenge 6

Confusion between contour and non-contour

changes caused by cluttered textures

changes correspond to object boundaries

Key Problems 7

Contour detection is usually formulated as a per-pixel classification problem

How to extract discriminative contour features?

How to learn a efficient contour classifier?

Perspective 8

Year 1990 2000 2004 2008

Learning Based

Contour Detection

Learning Based

Contour Detection

Edge Detection Edge Detection

Deep Learning

Based Contour

Detection

Deep Learning

Based Contour

Detection

8

2014

Outline 9

Contour Detection

Overview

Milestones Our Work

Discussions

Edge Detection 10

Use local differential template to compute gradient

121

000

121

FX

101

202

101

FY

22YXG

The original Sobel edge

Related Works 11

[1] J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 8(6):679–698, 1986 [2] R. Kimmel and A.M. Bruckstein (2003) "On regularized Laplacian zero crossings and other optimal edge integrators", International Journal of Computer Vision, 53(3) pages 225-243. [3] T. Lindeberg (1993) "Discrete derivative approximations with scale-space properties: A basis for low-level feature extraction", J. of Mathematical Imaging and Vision, 3(4), pages 349--376. [4] D. Ziou and S. Tabbone (1998) "Edge detection techniques: An overview", International Journal of Pattern Recognition and Image Analysis, 8(4):537–559, 1998 [5] H. Farid and E. P. Simoncelli, Differentiation of discrete multi-dimensional signals, IEEE Trans Image Processing, vol.13(4), pp. 496--508, Apr 2004

[1] J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 8(6):679–698, 1986 [2] R. Kimmel and A.M. Bruckstein (2003) "On regularized Laplacian zero crossings and other optimal edge integrators", International Journal of Computer Vision, 53(3) pages 225-243. [3] T. Lindeberg (1993) "Discrete derivative approximations with scale-space properties: A basis for low-level feature extraction", J. of Mathematical Imaging and Vision, 3(4), pages 349--376. [4] D. Ziou and S. Tabbone (1998) "Edge detection techniques: An overview", International Journal of Pattern Recognition and Image Analysis, 8(4):537–559, 1998 [5] H. Farid and E. P. Simoncelli, Differentiation of discrete multi-dimensional signals, IEEE Trans Image Processing, vol.13(4), pp. 496--508, Apr 2004

Learning Based Contour Detection 12

Learn a classifier to combine different features

Image Boundary Cues

Model

Pb

Brightness

Color

Texture

Cue Combination

Learning Based Contour Detection 13

Discriminative features

Spectral component (gPb - Arbeláez et al. PAMI 11)

Sparse code gradients (SCG – Ren&Bo, NIPS 12)

Learning Based Contour Detection 14

Efficient Detector

Structured Forest (SE – Dollar&Zitnick ICCV 13, PAMI 15)

Related Works 15

[1] D. Martin, C. Fowlkes, J. Malik. Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues. IEEE Trans. Pattern Anal. Mach. Intell. 26(5): 530-549 (2004) [2] P. Arbelaez, M. Maire, C. Fowlkes, J. Malik. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5): 898-916 (2011) [3] X. Ren. Multi-scale Improves Boundary Detection in Natural Images. ECCV (3) 2008: 533-545 [4] X. Ren, L. Bo. Discriminatively Trained Sparse Code Gradients for Contour Detection. NIPS 2012: 593-601 [5] J. Lim, C. Zitnick, P. Dollár. Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection. CVPR 2013: 3158-3165 [6] P. Dollár, C. Zitnick. Structured Forests for Fast Edge Detection. ICCV 2013: 1841-1848 [7] P. Dollár, Z. Tu, S. Belongie. Supervised Learning of Edges and Object Boundaries. CVPR (2) 2006: 1964-1971 [8] P. Arbeláez, J. Pont-Tuset, J. Barron, F. Marqués, J. Malik. Multiscale Combinatorial Grouping. CVPR 2014: 328-335

[1] D. Martin, C. Fowlkes, J. Malik. Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues. IEEE Trans. Pattern Anal. Mach. Intell. 26(5): 530-549 (2004) [2] P. Arbelaez, M. Maire, C. Fowlkes, J. Malik. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5): 898-916 (2011) [3] X. Ren. Multi-scale Improves Boundary Detection in Natural Images. ECCV (3) 2008: 533-545 [4] X. Ren, L. Bo. Discriminatively Trained Sparse Code Gradients for Contour Detection. NIPS 2012: 593-601 [5] J. Lim, C. Zitnick, P. Dollár. Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection. CVPR 2013: 3158-3165 [6] P. Dollár, C. Zitnick. Structured Forests for Fast Edge Detection. ICCV 2013: 1841-1848 [7] P. Dollár, Z. Tu, S. Belongie. Supervised Learning of Edges and Object Boundaries. CVPR (2) 2006: 1964-1971 [8] P. Arbeláez, J. Pont-Tuset, J. Barron, F. Marqués, J. Malik. Multiscale Combinatorial Grouping. CVPR 2014: 328-335

Deep Learning Based Contour Detection

16

Use deep networks to extract contour features and predict contour map

Contour Patch Contour Patch Background

Patch Background

Patch

Feature map Feature map

Related Works 17

[1] J. J. Kivinen, C. K. I.Williams, and N. Heess. Visual boundary prediction: A deep neural prediction network and quality dissection. In Proc. AISTATS, pages 512–521, 2014. [2] Y. Ganin and V. S. Lempitsky. N4-fields: Neural network nearest neighbor fields for image transforms. In Proc. ACCV, 2014. [3] W. Shen, X. Wang, Y. Wang, X. Bai, Z. Zhang. DeepContour: A Deep Convolutional Feature Learned by Positive-sharing Loss for Contour Detection. In Proc. CVPR, 2015. [4] G. Bertasius, J. Shi, L. Torresani. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection. In Proc. CVPR, 2015. [5] J.-J. Hwang, T.-L. Liu. Pixel-wise Deep Learning for Contour Detection. CoRR abs/1504.01989 (2015) [6] J.-J. Hwang, T.-L. Liu: Contour Detection Using Cost-Sensitive Convolutional Neural Networks. CoRR abs/1412.6857 (2014)

[1] J. J. Kivinen, C. K. I.Williams, and N. Heess. Visual boundary prediction: A deep neural prediction network and quality dissection. In Proc. AISTATS, pages 512–521, 2014. [2] Y. Ganin and V. S. Lempitsky. N4-fields: Neural network nearest neighbor fields for image transforms. In Proc. ACCV, 2014. [3] W. Shen, X. Wang, Y. Wang, X. Bai, Z. Zhang. DeepContour: A Deep Convolutional Feature Learned by Positive-sharing Loss for Contour Detection. In Proc. CVPR, 2015. [4] G. Bertasius, J. Shi, L. Torresani. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection. In Proc. CVPR, 2015. [5] J.-J. Hwang, T.-L. Liu. Pixel-wise Deep Learning for Contour Detection. CoRR abs/1504.01989 (2015) [6] J.-J. Hwang, T.-L. Liu: Contour Detection Using Cost-Sensitive Convolutional Neural Networks. CoRR abs/1412.6857 (2014)

Outline 18

Contour Detection

Overview

Milestones

Our Work Discussions

Our Work 19

DeepContour: A Deep Convolutional Feature Learned by Positive-sharing

Loss for Contour Detection

Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, Zhijiang Zhang

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015

Motivation 20

Why to apply CNN for contour detection?

Contour is hard to define

Contour data are sufficient for CNN training (millions

of local contour patches)

Problem Formulation 21

Given a color image patch 𝑥 ∈ ℝ𝑛×𝑛×3, our goal is to determine whether its center pixel is passed through by contours or not.

𝑥 ∈ ℝ𝑛×𝑛×3 → 𝑧 ∈ 0,1

Q: Is a good idea to directly use CNN as a blackbox to address this binary classification problem?

Non-contour Contour

Obstacle 22

The large variations in the contour shapes

Solution: Partitioning contour patches into compact clusters to convert the binary classification problem to a multi- class classification problem

Ref: J. J. Lim, C. L. Zitnick, and P. Dollár. Sketch tokens: A learned mid-level representation for contour and object detection. CVPR, 2013.

Obstacle 23

How to define the loss function?

Q: Is softmax a good choice?

Softmax function penalizes the loss of each class equally

The losses for contour versus non-contour should be emphasized

Solution: Adding a regularized term to focus on the end goal of binary classification

24

Data Preparation

Pre-cluster contour patches according to their contour shapes.

Assign a label 𝑦 to each contour patch 𝑥 according to the pre-cluster index 1,…𝐾 .

𝑥 ∈ ℝ𝑛×𝑛×3 → 𝑦 ∈ 0,1,…𝐾

Negative Positive

25

Method

CNN Architecture

Loss Function

Let (𝑎𝑗𝑖; 𝑗 = 1,…𝐾) be the output of unit 𝑗 in FC2 for a

image patch 𝑥𝑗(𝑖)

, the probability that the label is 𝑗 is

𝑝𝑗(𝑖)=exp(𝑎𝑙

(𝑖))

exp(𝑎𝑙(𝑖))𝐾

𝑙=0

26

Method

𝐽 = −1

𝑚 𝟏 𝑦 𝑖 = 𝑗 log 𝑝𝑗

𝑖

𝐾

𝑗=0

𝑚

𝑖=1

−1

𝑚 𝜆 𝟏 𝑦 𝑖 = 0 log 𝑝0

𝑖+ 𝟏 𝑦 𝑖 = 𝑗 log(1 − 𝑝0

𝑖)

𝐾

𝑗=1

𝑚

𝑖=1

Softmax loss

Positive-sharing loss, the loss for positive class is shared among each pre-clustered contour classes

27

Method

To apply standard back-propagation to optimize the parameters of the network

𝜕𝐽

𝜕𝑎0(𝑖)=1

𝑚𝜆 + 1 𝟏 𝑦 𝑖 = 0 𝑝0

𝑖− 1 + (𝜆 + 1) 𝟏 𝑦 𝑖 = 𝑗

𝐾

𝑗=1

𝑝0𝑖

𝜕𝐽

𝜕𝑎𝑙(𝑖)=1

𝑚𝜆𝟏 𝑦 𝑖 = 0 + 1 𝑝𝑙

(𝑖)− 𝟏 𝑦 𝑖 = 𝑙 − 𝜆 𝟏 𝑦 𝑖 = 𝑗

𝐾

𝑗=1

𝑝0𝑖𝑝𝑙𝑖

1 − 𝑝0𝑖

28

Method

CNN model validation

𝛾 ∈ [−1,1], measuring the discrimination of the learned model between positive and negative samples.

The learned features of FC1 will be fed into structured forest to perform contour detection.

𝛾 =1

𝑚 𝟏 𝑦 𝑖 = 0 − 𝟏 𝑦 𝑖 > 0 (𝑝0

𝑖− (1 − 𝑝0

𝑖))

𝑚

𝑖=1

29

Experimental results

Deep Feature Visualization

Positive-sharing loss

Local gradient

Experim

ental resu

lts

30

Results on BSDS500

31

Experimental results

Results on BSDS500

32

Experimental results

Results on NYUD

33

Experimental results

Cross Dataset Generalization

DeepContour ODS OIS AP

BSDS/BSDS .76 .78 .80

NYU/BSDS .72 .74 .77

BSDS/NYU .59 .60 .53

NYU/NYU .62 .63 .57

SE [11] ODS OIS AP

BSDS/BSDS .74 .76 .78

NYU/BSDS .72 .73 .76

BSDS/NYU .55 .57 .46

NYU/NYU .60 .61 .56

SCG [39] ODS OIS AP

NYU/NYU .55 .57 .46

gPb [2] ODS OIS AP

NYU/NYU .51 .52 .37

34

Experimental results

Parameter Discussion

Outline 35

Contour Detection

Overview

Milestones

Our Work

Discussions

Discussions 36

Speed

Caffe – per-patch mean subtraction is the bottleneck

Accuracy

Limitation may caused by the confusing labels

Discussions 37

Thank you! Q&A