scale and object aware image retargeting for thumbnail browsing
DESCRIPTION
《Scale and Object Aware Image Retargeting for Thumbnail》的PPT perillaroc@BITTRANSCRIPT
![Page 1: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/1.jpg)
Scale and Object Aware Image Retargeting for Thumbnail Browsing
Jin Sun, Haibin Ling
Temple University, Philadelphia, PA, 19122
By perillaroc
ICCV 2011
![Page 2: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/2.jpg)
Introduction
• Image browsing tasks
• Tiny thumbnails: a fixed small size
• Scaling
– bring difficulties in searching and recognition
• Automatic image retargeting methods
– target size is comparable to that of the original image
• Several important issues for thumbnail browsing
2012/6/19 2
![Page 3: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/3.jpg)
Thumbnail Browsing
• Thumbnail scales
– Studies have shown that scales can have
significant effects on human visual perception
• Object completeness
– Low-level gradient-based information
– NOW! The object-level completeness
• Structure smoothness
– The contamination caused by pixel removal
methods, e.g. seam carving
2012/6/19 3
![Page 4: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/4.jpg)
Scale and Object Aware Retargeting
• Thumbnail scales
– An image perceived by human vision system
– A new scale dependent saliency
• Object completeness
– integrating the objectness measurement recently
proposed by Alexe et al.
• Structure smoothness
– Use the thin-plate-spline(TPS) as warping function
– Use Cyclic seam Carving to guide the warping.
2012/6/19 4
![Page 5: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/5.jpg)
Scale and Object Aware Retargeting
• A continuous retargeting algorithm but uses discrete retargeting to guide the warping transformation.
2012/6/19 5
![Page 6: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/6.jpg)
Overview : Task Formulation
• Retargeting problem:
• Seam carving(SC)
• First get retargeted images to a target size that is comparable to the original image size.
• Then shrink the retargeted images to get the final thumbnails.
( , ( ))J F I S I
,
( , ), ( )
( , 1), ( )
sc
i j
I i j if j seam iF
I i j if j seam i
2012/6/19 6
![Page 7: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/7.jpg)
Overview : Framework Overview
• Scale and object aware image retargeting(SOAP):
• Warping function: thin-plate-spline (TPS)
• Landmark points : cyclic seam carving (CSC)
• Scale dependent saliency
• Object awareness
• Scale and object aware saliency
• SOAP
soF
( )scaleS I
( )O I
( ) ( ) ( )so scaleS I S I O I
( , ( ))so soJ F I S I
2012/6/19 7
![Page 8: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/8.jpg)
Flow chart of the proposed method
2012/6/19 8
![Page 9: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/9.jpg)
Scale-dependent Saliency
• Original image size
• Display size
• Projection size
• DPI(Dots Per Inch)
,d rpo
d p
s Dss s
DPI D
2012/6/19 9
![Page 10: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/10.jpg)
Scale-dependent Saliency
• How well such a system preserves the image information?
• We want to make the foreground object/ content/theme of the image as “clear” as possible
• According to the study, not all patterns in an image are recognizable by human
2012/6/19 10
![Page 11: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/11.jpg)
Scale-dependent Saliency • the perceived image, denoted as , can be derived as
• is the mean value of N-connected neighbors of pixel I(i,j)
• N is determined by display device specifications.
• κ and ρ are the lower and upper bounds(in cycles per degree) respectively of human visual acuity.
• κ and ρ define the limits at which the visual stimuli frequency becomes too low or too high to be recognized by human.
• We use κ = 0.0175 and ρ = 0.83.
pI
( , ), / [ , ]( ( , ))
( , ),
d
p
N
I i j if s DI I i j
I i j otherwise
( , )NI i j
2012/6/19 11
![Page 12: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/12.jpg)
Scale-dependent Saliency
• A pixel may become indistinguishable from its neighbors to a human observer.
• An image patch that was salient in the original image may not appear salient to a human observer.
• Scale dependent saliency
2012/6/19 12
![Page 13: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/13.jpg)
Scale-dependent Saliency
• First, the original image is scaled in homogeneous into the final thumbnail size, i.e. the display size, 60x60 pixels in our experiment.
• Then, the minimum recognizable pattern, denoted by
is determined by Eqn.4.
• Finally, the scale dependent saliency is defined as
where S(.) is the standard saliency (Itti 1998)
( ) ( )scale
pS I S I
( )scaleS I
ds
2012/6/19 13
![Page 14: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/14.jpg)
Scale-dependent Saliency • the minimum recognizable pattern
• In our monitor : 1680X1050@65Hz , 120 DPI
• is in average 0.009 inches.
• The size is approximately the distance between two pixel lines on the screen, i.e. N = 4 in Eqn.5.
• As a result, in the final thumbnail four adjacent neighbors of one image pixel patch with value differences in color space within certain threshold will be assigned their mean value, which means those pixels are unable to be distinguished by human.
• Differences : 50 in our experiment
ds
ds
2012/6/19 14
![Page 15: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/15.jpg)
Scale-dependent Saliency
2012/6/19 15
![Page 16: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/16.jpg)
Objectness Measure
• Preserve object completeness as much as possible
• Alexe et al. What is an object? (CVPR,2010).
– A novel objectness measure, which is trained to distinguish object windows from background ones.
– For a rectangular window w = (i,j,w,h) with a top-left corner at (i,j), width w and height h, its objectness is defined as the probability p(w) that w contains an object.
2012/6/19 16
![Page 17: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/17.jpg)
Objectness Measure
• First sample windows for an input image
• Then calculate the objectness map O as the average objectness reponse at each pixel
Where
• = 10000 in our experiments
wn1{ }w
i iW w
( , )
1( , ) ( )obj
w W i j w
O i j p w
,max ( , )i j O i j
wn
2012/6/19 17
![Page 18: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/18.jpg)
Scale and Object Aware Saliency
( , ) ( , )so scaleS S i j O i j
2012/6/19 18
![Page 19: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/19.jpg)
Cyclic Seam Carving • In many cases a seam has no choice but to cross objects due
to the original definition of seam.
• Discontinuous seams
• Discontinuous seam-carving for video retargeting. CVPR, 2010
• Problem: object-structure damage, e.g. pixel shift problem
2012/6/19 19
![Page 20: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/20.jpg)
Cyclic Seam Carving
• Cyclic Seams: warp the image
2012/6/19 20
![Page 21: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/21.jpg)
Cyclic Seam Carving
• Energy function
where ρ is the weight and set to 0.3 .
• The improved energy is then combined with the CSC algorithm to provide landmark point pairs needed for estimating TPS warping.
(1 ) so
scale scE E S
2012/6/19 21
![Page 22: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/22.jpg)
Cyclic Seam Carving
2012/6/19 22
![Page 23: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/23.jpg)
Image Warping Function
• Many discrete retargeting methods generate excellent results in general but they sometimes create serious artifacts when the target has a size much smaller than the input.
• We combine a continuous warping model with a discrete retargeting guidance.
• Thin-plate-spline(TPS) as our warping function
2012/6/19 23
![Page 24: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/24.jpg)
Thin-plate-spline(TPS)
• Landmark points and
, where p is mapped to q.
• the TPS transformation T is defined as the transformation from P to Q that minimizes the regularized bending energy E(f) defined as
• The TPS warping is defined as
2 , 1,2, ,i lP p i n
2 , 1,2, ,i lQ q i n
2
2 2 22 2 2
2 2
( ) ( )
( ) 2( ) ( )
i i
i
E f q f q
f f fdxdy
x x y y
argmin ( )fT E f
2012/6/19 24
![Page 25: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/25.jpg)
TPS Warping Function
• Landmark point pairs P and Q is derived from the CSC retarget algorithm
• A two-way solution
• First, We sample randomly a landmark set b P (Fig. 6(a)) from original image and then trace their shifting in the CSC process.
2012/6/19 25
![Page 26: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/26.jpg)
TPS Warping Function
• Then, point set Q is re-sampled uniformly on the target image, which is generated by CSC.
• Finally, a sample set P is generated by mapping Q to the original image using warping estimated by Q and P.
2012/6/19 26
![Page 27: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/27.jpg)
Quantitative Experiments
• To find the thumbnail that matches the description
• From 10 x 10 image thumbnails
• Different methods
– Scaling(SL)
– Seam carving (SC)
– Improved seam carving (ISC)
– The proposed SOAR algorithm (SOAR)
2012/6/19 27
![Page 28: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/28.jpg)
Quantitative Experiments
2012/6/19 28
![Page 29: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/29.jpg)
Quantitative Experiments
2012/6/19 29
![Page 30: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/30.jpg)
Qualitative Experiments
• In general our method performs the best regarding the (foreground) object size in the thumbnails.
2012/6/19 30
![Page 31: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/31.jpg)
RetargetMe
2012/6/19 31
![Page 32: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/32.jpg)
Limitations
• Saliency distribution is scattered
• Object is to big
2012/6/19 32
![Page 33: Scale and object aware image retargeting for thumbnail browsing](https://reader034.vdocuments.site/reader034/viewer/2022042623/54b517634a79596f458b460b/html5/thumbnails/33.jpg)
Conclusion
• A scale and object aware image retargeting method for thumbnail browsing.
• Several new techniques
– Scale dependent saliency
– Objectness
– Cyclic seam carving
– A TPS-based continuous warping model
2012/6/19 33