MODELING VISUAL CLUTTER PERCEPTION USING PROTO-OBJECTS

Chen-Ping Yu

Prof. Dimitris Samaras

Prof. Greg Zelinsky

Introduction• The goal: model human visual clutter perception.

• Visual clutter: A “confused collection”, or a “crowded disorderly state”. Increasing visual clutter.

• Set Size Effect: search performance decreases as set size increases. How to quantify set size?

Introduction• What are proto-objects?

• Regions of locally similar features, they can be objects, object parts, or just pieces that come together to form objects.

• What is the proto-object clutter model?• Segments an image into proto-objects, use the normalized number

of proto-objects as the clutter measure of the image.

Stimuli• 90 800x600 images of real world contexts

• Selected from the SUN09 Database• 6 object-count groups, each contains 15 images• Human labeled objects are provided with SUN09

31 32 33

36 37 39

31~40 objects

15 images

51 52 53

55 57 58

51~60 objects

15 images

3 5 7

7 9 10

1~10 objects

15 images

90 images total

Stimuli

17

48

3

57

Behavioral method• Subjects

• 15 human subjects age from 18 to 32

• Method• Rank order the 90 images from least to most cluttered• Using a Matlab GUI• Participants were told to use their own definition of clutter

• Practice• 12 practice images prior to actual testing

Behavioral method• 2 displays were used

• Images were shown at random

• Bottom monitor subtended a visual angle of 27° x 20°

• Had the option to correct the ordering

• A experiment lasted roughly 45~60 min

• Average pair-wise rater agreement: ρ = 0.692

Computational method• 1. superpixel preprocessing

• K = 600• Need to merge the resulting over-segmentation

Computational method• 2. Mean-shift clustering

Computational method• 2. Mean-shift clustering, more examples

Computational method• 2. Mean-shift clustering in HSV color space

• Median color of each superpixel

Computational method• 3. Merge neighboring superpixels that belong to the same

color cluster

600 superpixels 207 proto-objects (0.345)

Computational method• Proto-object visualization

• Fill each proto-object using the median of the member-pixel colors

Results• Spearman’s rank order correlation

• Ρ = 0.814, p < 0.001

Results• Robust to different parameter/color space settings

• Each correlation is computed using the optimal MS bandwidth

Results• Comparing to other clutter models

Results• More visualized proto-object results

Some further experiments• Does visual clutter perception change when viewing

images of different sizes?

• Experiment: Large images vs small images• Same 90-image dataset, large images = original 800x600 size;

small images = quarter size (200x150)

• Same behavioral setting• 12 practice images• Same Matlab GUI

Further experiments• Small images subtended a

visual angle of 6.75° x 5 °

• 20 undergraduate students from SBU

• Followed the same procedure as the large-image setup

• Average inter subject correlation: 0.58

Further experiments• Highlights

• Human’s small image rating vs large image rating: ρ = 0.953..!• Proto-object model’s small image correlation: ρ = 0.852

Further experiments• Comparing with other clutter models

• Proto-object model stayed the most consistent

Conclusion• Number of object (set size) is a poor predictor to visual

clutter

• Set size may be better quantified/represented by proto-objects• All segmentation-based methods outperformed the feature/non-

segmentation based models

• Can proto-object’s spatial density predict search performance, and/or number estimation?

• Can proto-object’s spatial distribution predict gaze?