Context-Aware Saliency Detection

Stas Goferman, Lihi Zelnik-Manor, Ayellet TalCVPR,2010

Outline

Introduction Detection of context-aware saliency Results Application Conclusion

Introduction(1)

Saliency identify fixation points?

This type of saliency is important for understanding human attention as well as for specific applications such as auto focusing.

Original image Saliency map Fixation points

Introduction(2)

Saliency identify the dominant object ?

Original image Saliency map Itti’s By labeler

This type of saliency is useful for several high-level tasks, such as object recognition or segmentation.

Introduction(3) Context -aware

For some application ,like summarization of a photo collection ,and retargeting…maintain the essential regions of the image is important.

Introduction(4)

Here , introducing a new type of saliency algorithm contain not only the prominent objects but also the parts of the background that convey the context.

The algorithm follows four psychological principles of human visual attention.

We demonstrate the applicability of two application, image retargeting and summarization.

Outline

Introduction Detection of context-aware saliency Results Application Conclusion

Principles of context-aware saliency

1. Local low-level considerations, including factors such as contrast and color. the distinctive color and other features area should obtain high attention.

2. Global considerations, which suppress frequently occurring features, while maintaining features that deviate from the norm. redundant information should be suppressed and popping up the novelty part.

Principles of context-aware saliency

3. Visual organization rules, which state that visual forms may possess one or several centers of gravity about which the form is organized. the salient pixels should be grouped together, and not spread all over the image.

4. High-level factors, such as human faces. implemented as post-processing.

Principles of context-aware saliency

(a)Local[24] (b)Global[7] (c)Local-global[13] (d)Context-aware

Comparing different approaches to saliency

Local-global single-scale saliency

1.

2.

We should not, however, look at an isolatedpixel, but rather at its surrounding patch.

Let dcolor(pi , pj) be the Euclidean distance between the vectorized patches pi and pj in CIE L*a*b colorspace, dposition(pi , pj) be the Euclidean distance between thepositions of patches pi and pj

It suffices to consider the K most similar patches

The single-scale saliency value of pixel i at scale r is defined as left . {qk} k=1 to K , K = 64 in our experiments

Multi-scale saliency enhancement

3.

4.

Background pixels (patches) are likely to have similar patches at multiple scales.

For a patch pi of scale r, we consider as candidate neighbors all the patches in the image whose scales are Rq = {r ,1/2r ,1/4r} .

Let R denote the set of patch sizes to be considered for pixel i.The saliency at pixel i is taken as the mean of its saliency at different scales

The larger Si is, the more salient pixel i is and the larger is its dissimilarity to the other patches.

Including the immediate context

1: A pixel is considered attended if its saliency value exceeds a certain threshold( Si > 0.8 in the examples

shown in this paper).

2: The saliency of a pixel is redefined as

Let dfoci(i) be the Euclidean positional distance between pixel i and the closest focus of attention pixel, normalized to the range [0,1]

Steps

The steps of our saliency estimation algorithm

High-level factors

Final , face detection algorithm

Si = Si , if Si > face(i)

face(i) , otherwise

face detection algorithm of [23], which generates 1 for face pixels and 0 otherwise.

modified

[23] P. Viola and M. Jones. Rapid Object Detection Using a Boosted Cascade of Simple Features. In CVPR, 2001.

Outline

Introduction Detection of context-aware saliency Results Application Conclusion

Salient results (1)

From left to right input , result of [24] , result of [7], our result .

Compare to other methods with three cases image.

(a) a single object over an uninteresting Background.

(b) the immediate surroundings of the salient object is also salient.

(c) complex scenes.

[24] D.Walther and C. Koch. Modeling attention to salient proto objects. Neural Networks, 19(9):1395–1407, 2006.[7] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR, pages 1–8, 2007.

Salient results (2)

[13] T. Liu, J. Sun, N. Zheng, X. Tang, and H. Shum. Learning to Detect A Salient Object. In CVPR, 2007.

Comparing our saliency results with [13]

Top: Input images.

Middle: The bounding boxes obtained by [13] capture a single main object.

Bottom: Our saliency maps convey the story .

Outline

Introduction Detection of context-aware saliency Results Application Conclusion

Image retargeting

Image retargeting aims at resizing an image by expanding or shrinking the non-informative regions.Distortions, if and when introduced, will exist only in regions of lower significance.

Input Saliency of [19] Our saliency Results of [19] Our result Figure 9. Seam carving of 100 ”vertical“ lines. The salient objects are distorted by [19] in contrast to our results.

[19]M. Rubinstein, A. Shamir, and S. Avidan. Improved seam carving for video retargeting. ACM Trans. on Graphics,27(3), 2008.

Summarization through collage creation

this technique consists of three stages

Compute saliency maps of the images

Extract regions-of-interest(ROI) by considering both saliency and image edge information

Assemble the non-rectangular ROIs ,allowing slight overlaps.

[4]S. Goferman, A. Tal, and L. Zelnik-Manor. Puzzle-like collage. Computer Graphics Forum (EUROGRAPHICS), 29, 2010.

Summarization through collage creation

Summarization of a trip to LA using 14 images.

(a)The saliency maps of the input images

(b)The collage summarization

Outline

Introduction Detection of context-aware saliency Results Application Conclusion

Conclusion

This paper proposes a new type of saliency – context aware saliency based on four principles observed in the psychological literature which detects the important parts of the scene.

In the future we intend to learn the benefits of this saliency in more applications, such as image classification and thumbnailing.

Appendix Global methods : Saliency Detection: A Spectral Residual Approach [7]

H(Image) = H(Innovation) + H(Prior Knowledge)

R(f) = L(f) − A(f)

A(f) = hn (f) L(f)∗