design and perceptual validation of performance measures for salient object segmentation vida...

Design and Perceptual Validation

of Performance Measures for Salient Object Segmentation

Vida Movahedi, James H. Elder

Centre for Vision Research

York University, Canada

Evaluation of Salient Object Segmentation

Centre for Vision Research, York University2

Source: Berkeley Segmentation Dataset

Evaluation of Salient Object Segmentation


How do we measure success?

Existing literature Salient object segmentation

[Liu07, Zhang07, Park07, Zhuang09, Achanta09, Pirnog09, …]

Evaluation of salient object segmentation algorithms [Ge06,?]

Evaluation of segmentation algorithms [Huang95, Zhang96, Martin01, Monteiro06, Goldmann08,

Estrada09]


Contributions


Analysis of previously suggested measures

Contour Mapping Measure (CM) Order-preserving matching

A new dataset of salient objects (SOD)

Psychophysics experiments Evaluation of above measures

Matching paradigm in Precision and Recall measures

Evaluation measures in literature


Region-based error measures Based on false positive/ false negative pixels [Young05], [Ge06], [Goldmann08], ...

Boundary-based error measures Based on distance between boundaries [Huttenlocher93], [Monteiro06], ...

Mixed measures Based on distance of misclassified pixels to the

boundaries [Young05], [Monteiro06], ...

Region-based error measures[Young05], [Ge06], [Goldmann08], ...


A and B two boundaries RA the region corresponding to a boundary A and |

RA| the area of this region,

BA

BA

RR

RRBARI

1),(BA

BAB

BA

BAA

RR

RRR

RR

RRR

||||

False Negatives

False Positives

Not sensitive to shape differences

Boundary-based error measures[Huang95],[Huttenlocher93], [Monteiro06], ...


A and B two boundaries Distance of one point a on A from B is

Hausdorff distance:

Mean distance:

),(min)( badadBb

B

)(max),(maxmax),( bdadBAHD ABb

BAa

)(mean),(meanmean),( bdadBAMD ABb

BAa

Not sensitive to shape differences

a

Penalizing the over-detected and under-detected regions by their distances to intersection

Mixture error measures [Young05], [Monteiro06], ...


fpfn N

kkB

fp

N

jjA

fndiag

qdN

pdND

BAMM11

)(1

)(1

2

1),(

False Negatives

False Positives

Not sensitive to shape difference

Another example


Different shapes with low errors

Comparing two boundaries


The two boundaries need to follow each other Thus it is not sufficient to map points to the

closest point on the other boundary The ordering of mapped points must be preserved

B ASmall False

Negative Region

Small False Positive Region

The order of mapped points on the two boundaries must be monotonically non-decreasing.

Allowing for different levels of detail: One-to-one Many-to-one One-to-many

Order-preserving Mapping

nmjibaba njmi then and ,If


Contour Mapping Measure


Given two contours A=a1a2..an and B=b1b2..bm, Find the correct order-preserving mapping

Contour mapping error measure:

Average distance between matched pairs of points

Bimorphism [Tagare02]

Elastic Matching [Geiger95, Basri98, Sebastian03, ..]

A dynamic programming implementation to find the optimum mapping Closed contours point indices are assigned cyclically

Based on string correction techniques [Maes90]

Complexity: if m<n and m, n points on two boundaries

Contour Mapping Measure


)log( mnmO

Contour Mapping Example


Ground Truth Boundary

Algorithm Boundary

Matched pairs shown as line segments

CM= average length of line segments

connecting matched pairs

Contour Mapping Measure Order- preserving mapping avoids problems

experienced by other measures


SOD: Salient Object Dataset


A dataset of salient objects Based on Berkeley Segmentation Dataset

(BSD) [Martin01]

300 images 7 subjects

1

1

1

1

1

Source: Berkeley Segmentation Dataset Available in SOD

Psychophysical experiments


Which error measure is closer to human judgements of shape similarity?

9 subjects 5 error measures Regional Intersection (RI) Mean distance (MD) Hausdorff distance (HD) Mixed distance (MM) Contour Mapping (CM)

Psychophysical Experiments

Experiment 1 - SOD

Reference & test shapes all from SOD

Experiment 2 - ALG

Reference from SOD, test shapes algorithm-generated


Reference: Human

segmentation

Test cases:

Algorithm-

generated

Test cases: Human

segmentations

Reference: Human

segmentation

Agreement with Human Subjects Human subject chooses Left

or Right

An error measure M also chooses Left or Right, based on their error w.r.t. the reference shape

If M chooses the same as the human, it is a case of agreement

Human-Human consistency: defined based on agreement between human subjects


Left Right

Reference

Psychophysical Experiments


Experiment 1- SOD

Reference & tests shapes all

from SOD

Experiment 2 - ALG

Reference from SOD, test shapes algorithm-

generated

RI: region intersection, MD: mean distance, HD: Hausdorff distance, MM: mixed measure, CM: contour mapping

Precision and Recall measures


For algorithm boundary A and ground truth boundary B

Precision: proportion of true positives on A

Recall: proportion of detected points on B

Martin’s PR (M-PR)[Martin04] Minimum cost bipartite matching, cost proportional to

distance

Estrada’s PR (E-PR)[Estrada09] ‘No intervening contours’ and ‘Same side’ constraints

Contour Mapping PR (CM-PR) Order-preserving mapping

||

),matched(

A

BA

||

),matched(

B

AB

Matching paradigm in Precision/Recall


Experiment 1- SOD

Reference & test shapes all

from SOD

Experiment 2 - ALG

Reference from SOD, test shapes algorithm-

generated

Summary


Analysis of available measures for evaluation of salient object segmentation algorithms

A new measure- contour mapping measure (CM) Code available online: http://elderlab.yorku.ca/ContourMapping

A new dataset of salient objects Dataset available online: http://elderlab.yorku.ca/SOD

Psychophysical Experiment CM has a higher agreement with human subjects

Order-preserving matching paradigm in Precision/Recall analysis Code available online: http://elderlab.yorku.ca/ContourMapping


Thank You!

design and perceptual validation of performance measures for salient object segmentation vida...

Documents

boundaries centre

mapping centre

york university12 slide

york university4 slide

evaluation measures

literature centre

canada slide

york university7