model comparison and challenges ii compositional bias of salient object detection benchmarking
DESCRIPTION
Model comparison and challenges II Compositional bias of salient object detection benchmarking. for the Crash Course on Visual Saliency Modeling: Behavioral Findings and Computational Models CVPR 2013. Xiaodi Hou K-Lab, Computation and Neural Systems California Institute of Technology. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/1.jpg)
Model comparison and challenges IICompositional bias of salient object detection benchmarking
Xiaodi HouK-Lab, Computation and Neural Systems
California Institute of Technology
for the Crash Course on Visual Saliency Modeling:Behavioral Findings and Computational Models
CVPR 2013
![Page 2: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/2.jpg)
Schedule
![Page 3: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/3.jpg)
On detecting salient objects• Learning to Detect A Salient Object [Liu et. al., CVPR 07]• Frequency-tuned Salient Region Detection [Achanta et. al., CVPR 09]
![Page 4: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/4.jpg)
The progress!• Some top performers:
– [PCA] What makes a patch distinct [Margolin et. al., CVPR 13]– [SF]Saliency filters [Perazzi et. al., CVPR 12]:
• F-Measure: 0.84– [GC]/[GC-seg]Global contrast-based salient region detection [Cheng et. al., CVPR 11]
• F-Measure: 0.75– [FT] Frequency Tuned Salient Region Detection [Achanta et. a.l., CVPR 09] :
• 0.65 by [Achanta et. al., CVPR 09].
Image from [Perazzi et. al., CVPR 2012]
![Page 5: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/5.jpg)
The progress?• Salient objects in PASCAL
VOC?– 850 images from VOC 2013
validation set.– Intersection of main challenge
and segmentation challenge.– Answers more questions:• Where is your algorithm (in
salient object detection)?• Where is salient object detection
(in computer vision).
![Page 6: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/6.jpg)
The progress
• FT: 0.28• GC: 0.39• SF: 0.35• PCA: 0.40• GC-seg: 0.38
55% performance
drop!!
![Page 7: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/7.jpg)
The arguments
• No!!These objects are not salient!
• Our algorithm works on images with salient objects only!
![Page 8: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/8.jpg)
The paradox of salient object detection
But hey, what is a “salient object”?
![Page 9: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/9.jpg)
COMPOSITIONAL BIAS
![Page 10: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/10.jpg)
Before we proceed…
• Google Image Search: “science”– Rutherford atomic model (9)– Test tubes (10)– Microscopes (4)– Double helix (3)– Old guys with crazy hair and glasses (3)
Stereotypes of science are not sciences!
![Page 11: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/11.jpg)
How to compose a biased salient object detection dataset
Decide to build a new salient object dataset!
So what is saliency?
Searching for unambiguous examples of saliency…
Found one! Add to my dataset!
Job done! Let other people play with my
dataset!
![Page 12: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/12.jpg)
The compositional bias
• Compositional bias: Biases introduced during the composition of a dataset:– Exaggerating on stereotypical attributes.• Limited variability in positive samples.• Lack of negative samples at all.
Unlike datasets in machine learning, where the dataset is the world, computer vision datasets are supposed to be a representation of the world.
---- [Torralba and Efros: Unbiased look at Dataset bias]
![Page 13: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/13.jpg)
Compositional bias: the statistics
• Object number
![Page 14: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/14.jpg)
Compositional bias: the statistics
• Object eccentricity
![Page 15: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/15.jpg)
Compositional bias: the statistics• Global foreground and background contrast
![Page 16: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/16.jpg)
Compositional bias: the statistics• Local foreground/background contrast (contour strength)
![Page 17: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/17.jpg)
TOWARDS A BETTER SALIENT OBJECT DATASET
![Page 18: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/18.jpg)
The new project
• Build a salient object detection dataset from a good object detection dataset (e.g. PASCAL VOC).
Let the eye fixations pick up those salient objects!
![Page 19: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/19.jpg)
Data collection (in process)
• SR Research EyeLink 1000• 2-sec viewing time.• “Free-viewing” instruction (will mention it later).• 3 subjects (more subjects on the way).
We will release the dataset very soon!
![Page 20: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/20.jpg)
What makes an object salient• Unit conversion:– From fixation maps– To object fixation score• sum of blurred fixation map intensity within the object
mask.
![Page 21: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/21.jpg)
Object size and saliency
• Large objects attract more fixations.
• Small objects receive denser fixations.
![Page 22: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/22.jpg)
Object size and saliency
![Page 23: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/23.jpg)
Objects, salient objects, and the most salient objects
• Salient objects:– Fixation score higher than
mean (67.3% objects).• Most salient objects:– Fixation score higher than
mean*2 (27.8% objects).
Image with fixation Object labeling Salient objects Most salient object(s)
![Page 24: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/24.jpg)
Salient objects and salient object detection
• Guess how does the algorithms perform on “salient objects” and “most salient objects”?
On all objects:• FT: 0.28• GC: 0.39• SF: 0.35• PC: 0.38
![Page 25: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/25.jpg)
Testing on salient objects
Salient objects on PASCAL VOC
60% performance
drop!!
• FT: 0.22• GC: 0.35• SF: 0.31• PCA: 0.38• GC-seg: 0.39
![Page 26: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/26.jpg)
Testing on most salient objects
Most salient objects on PASCAL VOC
• FT: 0.10• GC: 0.20• SF: 0.15• PCA: 0.26• GC-seg: 0.23
79.8% performance
drop!!
![Page 27: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/27.jpg)
Something is wrong, seriously!
![Page 28: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/28.jpg)
DISCUSSIONS
![Page 29: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/29.jpg)
The role of saliency in a visual system
• Bad performance because of boundary detection?
• Bad performance because of unpredictability of human “free will”?
![Page 30: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/30.jpg)
Saliency as an oracle
• Oracle selecting the best segment– CPMC: 78% from 154 segments– gPB: 61% from 1286 segments
* coverage = intersect/union
![Page 31: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/31.jpg)
Saliency and tasks
• Build a salient object detection dataset from an egocentric object dataset.
• Let the eye-fixation speaks
Eye TrackerForward-looking Camera
Learning to recognize daily actions using gaze, [Fathi et. al. ECCV 12]
![Page 32: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/32.jpg)
What makes an object salient?Task
ObjectSaliency
• Object in egocentric actions• Fixated object ==
Manipulated object?
![Page 33: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/33.jpg)
THANKS
![Page 34: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/34.jpg)
Acknowledgement
• Joint work with Yin Li @ Gatech.
• Special thanks to Nathan Faivre for his kind help on eye tracking.
![Page 35: Model comparison and challenges II Compositional bias of salient object detection benchmarking](https://reader036.vdocuments.site/reader036/viewer/2022062814/568167e4550346895ddd4e90/html5/thumbnails/35.jpg)
Open discussions