contextual in-image advertising - clemson universityjzwang/ustc11/mm08...4 motivation •online...
TRANSCRIPT
Contextual In-Image Advertising
Tao Mei, Xian-Sheng Hua, Shipeng Li
Microsoft Research Asia
1ACM Multimedia 2008
ACM Multimedia 2008 2
3
Outline
• Motivation
• ImageSense
– Framework
– Problem formulation
– Global and local relevance
– Problem optimization
• Experiments
• Summary
ACM Multimedia 2008
Crawler
website
VB-1
VB-2-1
VB-2-2
VB-2-1
•Position: (100, 200)
•Width: 250
•Height: 500
•URL: http://msra-
va/stars/Images/Yao
MingPlay.jpg
Surrounding Text:Yao Ming, NBA, Houston Rockets,
Basketball, Sports, …
Image:
Vision-based Page Segmentation and Image Block Detection
Image Filtering
Web Page Text
Block-based Surround. Text
Expansion Text
High-level Concepts
ConceptDetector
Global + Local Textual Relevance
ImageSalience
Detection
Ad Insertion Point Detection
TextualRelevanceMatching
AdsAd Rank
List
…
Relevance Optimization
Visual RelevanceMatching
AdReranking:Ad-BlockMatching
4
Motivation
• Online images
ACM Multimedia 2008
• 2.8 bln image uploads
by 30 mln members till
2008 Sept.
• 2,530 image uploads
per min in 2008 Oct.
• 50 bln digital photos
taken in 2007, 60 bln
by 2011
• Online advertising“… grow to $25.9 billion this year, from $21.1 billion last year, and hit $30 billion in 2009.”
• Image-driven advertising
From the perspective of industry
• Conventional advertising: treat image advertising as text advertising
– Match ads with images only based on textual info.
– Insert ads at a fixed position
ACM Multimedia 2008 5
Ads are not always contextually relevant to image content
Preserved ad blocks are intrusive and break page structure
6
From the perspective of research
• Webpage advertising
– Ad keyword selection [Yih, WWW’06; Shen, SIGIR’06]
– Ad relevance matching• Keyword-targeted ads (paid search ads) [Mehta, JACM’07]
• Content-targeted ads (contextual ads) [Rib, SIGIR’05; Broder, SIGIR’05]
• User-targeted ads (audience intelligence) [Richardson, WWW’07]
• Video advertising
– VideoSense [Mei, MM’07]
– vADeo [Srinivasan, MM’07]
• Image advertising (to be investigated)
– Delivering ads inside images during image buffering [Li, MM’08]
ACM Multimedia 2008
7
What we want to argue
• Images are more powerful and effective information carriers– carry more information
– more attractive than text
– shown faster than video
• Ads are to be embedded in-image– no predefined ad blocks required
– ads promoted by salient appearance
• Ads are to be contextually relevant– web pages are too much or too few to describe image contents
– image content and its surrounding text are more precise
ACM Multimedia 2008
>>
8
What we want to propose
• ImageSense: a contextual in-image advertising platform– automatically matches ads to image content
– finds nonintrusive ad insertion positions within images
– seamlessly embeds relevant ads based on these positions
ACM Multimedia 2008
9
Outline
• Motivation
• ImageSense
– Framework
– Problem formulation
– Global and local relevance
– Problem optimization
• Experiments
• Summary
ACM Multimedia 2008
ImageSense framework
ACM Multimedia 2008 10
Crawler
website
VB-1
VB-2-1
VB-2-2
VB-2-1
•Position: (100, 200)
•Width: 250
•Height: 500
•URL: http://msra-
va/stars/Images/Yao
MingPlay.jpg
Surrounding Text:Yao Ming, NBA, Houston Rockets,
Basketball, Sports, …
Image:
Vision-based Page Segmentation and Image Block Detection
Image Filtering
Web Page Text
Block-based Surround. Text
Expansion Text
High-level Concepts
ConceptDetector
Global + Local Textual Relevance
ImageSalience
Detection
Ad Insertion Point Detection
TextualRelevanceMatching
AdsAd Rank
List
…
Relevance Optimization
Visual RelevanceMatching
AdReranking:Ad-BlockMatching
1. “relevance” - how to select ads? vision-based page segmentation multimodal relevance matching
2. “position” - where to insert ads? image saliency detection
Three key Problems Solutions
3. “appearance” - how to display ads? auto-animation seamless blending and transition
Objective: maximize overall relevance while minimize intrusiveness
• global textual relevance
• local textual relevance
• local content relevance
Problem formulation
ACM Multimedia 2008 11
image block:
ad:
traditional text ads proposed contextual ads
Preprocess:Page segmentation & Image block detection
ACM Multimedia 2008 12
VIPS
• Deng Cai, Shipeng Yu, Ji-Rong Wen and Wei-Ying Ma. "VIPS: a Vision-based Page Segmentation Algorithm,” MSR-TR-2003-79,2003.
VB-1
VB-1-2
VB-1-2-1
VB-1-1
VB-1-2-1
VB-1-2-1-1 VB-1-2-1-2
Web page
VB-1-2-1-2-1 VB-1-2-1-2-2
VB-1-2-1-1-1-1 VB-1-2-1-2-2-2
VB-1
VB-1-1
VB
-1-2
-1-2
VB-1-2-1-1
VB-1-2-1-2-1
VB-1-2-2
VB-1-2-1-2-2-2
VB-1-2-1-2-2-1
Global and local textual relevance
ACM Multimedia 2008 13
Keywords:
Pyramids of Giza, Egypt
Description:
the most famous monuments of
ancient Egypt. These massive
stone structures were built around
4500 years ago on a rocky desert
plateau close to the Nile. But the
intriguing Egyptian pyramids were
more than just tombs for kings.
The mysteries surrounding their
symbolism, design and purpose
have inspired passionate debate.
It is likely that many of these
mysteries will never be solved ...
Original Surrounding Text
Outdoor, Mountain, Person
Concept Text
Image
earth, shape, circle, aztec, food,
pyramid, geography, triangle,
egypt, ufo, rainbow, sunrise,
mummy, prism, dinosaur, empire,
khufu, taba, cheops, great
pyramid, dahab, aswan, acient,
valley of the kings, pyrimid,
tansik, pangea, saqqara,
hurghada, greece, egyptian,
africa, dubai, cyprus, jamaica,
ghana
Expansion Text
VIPS
T = {original text, expansion text, concept text}
Global textual relevance:
Local textual relevance:
BM25 similarity:
Local content relevance
ACM Multimedia 2008 14
1.0 0.8 0.6 0.8 1.0
1.0 0.8 0.6 0.8 1.0
0.8
0.8
0.8
0.8
0.6 0.6
0 0 0
0 0 0
0 0 0
Yu-Fei Ma, Hong-Jiang Zhang, “Contrast-based image attention analysis by using fuzzy growing,” ACM Multimedia, 2003.
original image Bi image saliency map Si weight map Wi
sibi
Local content relevance:
f: visual feature
non-salience visual similarityW
ACM Multimedia 2008
Problem optimization
15
Nonlinear 0-1 integer programming problem (NIP):
select candidate image
blocks with least saliency
select candidate ads with
best textual relevance
associate image blocks with
ads with best relevance
Computational complexity:
How to apply ImageSense
ACM Multimedia 2008 16
Keyword: Ming Yao, sports, …
• Website• Webpage• WebImage
Crawling/Segmentation
Image Analyzer
Image-AD Matching
Ad & ad position
AdDatabase
Ad
Experiments - data
• Advertisements– 7,285 unique ad product logos
– annotated by 20 subjects
– 32,480 unique ad words
• Online images– 382,371 images from www.tango.msra
– 200,000 images from Flickr
• Evaluation– 1,100 ad triggering pages
• 100 web pages from major news sites
• 1,000 images searched by top 100 image queries
– 86.74 words in average per page/image
ACM Multimedia 2008 18
Experiments - methodology
• Different advertising strategiesI. global textual relevance (traditional advertising)
II. global and local textual relevance without concept text
III. global and local textual relevance with concept text
IV. textual relevance and content relevance with equal weights
V. textual relevance and content relevance, with different weights from IV(wg=0.3, wl=0.5, wc=0.2)
ACM Multimedia 2008 19
Experiments - methodology
• Different advertising strategiesI. global textual relevance (traditional advertising)
II. global and local textual relevance without concept text
III. global and local textual relevance with concept text
IV. textual relevance and content relevance with equal weights
V. textual relevance and content relevance, with different weights from IV(wg=0.3, wl=0.5, wc=0.2)
ACM Multimedia 2008 20
I
Experiments - methodology
• Different advertising strategiesI. global textual relevance (traditional advertising)
II. global and local textual relevance without concept text
III. global and local textual relevance with concept text
IV. textual relevance and content relevance with equal weights
V. textual relevance and content relevance, with different weights from IV(wg=0.3, wl=0.5, wc=0.2)
ACM Multimedia 2008 21
II: Rl without concept text
Experiments - methodology
• Different advertising strategiesI. global textual relevance (traditional advertising)
II. global and local textual relevance without concept text
III. global and local textual relevance with concept text
IV. textual relevance and content relevance with equal weights
V. textual relevance and content relevance, with different weights from IV(wg=0.3, wl=0.5, wc=0.2)
ACM Multimedia 2008 22
III: Rl with all kinds of texts
Experiments - methodology
• Different advertising strategiesI. global textual relevance (traditional advertising)
II. global and local textual relevance without concept text
III. global and local textual relevance with concept text
IV. textual relevance and content relevance with equal weights
V. textual relevance and content relevance, with different weights from IV(wg=0.3, wl=0.5, wc=0.2)
ACM Multimedia 2008 23
IV: wg = wl = wc = 1/3
Experiments - methodology
• Different advertising strategiesI. global textual relevance (traditional advertising)
II. global and local textual relevance without concept text
III. global and local textual relevance with concept text
IV. textual relevance and content relevance with equal weights
V. textual relevance and content relevance, with different weights from IV (wg=0.3, wl=0.5, wc=0.2)
ACM Multimedia 2008 24
V: wg = 0.3, wl = 0.5, wc = 0.2
25
(a) images with poor tags (3.01) (b) images with rich tags (17.93)
(c) web pages (789.8) (d) overall (1,100 triggering pages)
Evaluation on Ad Relevance
• Observations– ImageSense outperforms traditional advertising, especially for the
images/pages with rich tags
– Surrounding texts and concept texts improve ad relevance
– Local content relevance is a tradeoff between ad relevance and user viewing experience
• Computational time– Online ad matching: < 0.1 second
– Image processing (800×600 pixel): < 0.1 second
– Overall: < 0.2 second
ACM Multimedia 2008 26
Evaluation on User Experience
• 15 subjects involved– 8 female and 7 male undergraduate/graduate students
– with different background
• Each gives a satisfaction score (1-5)– Ad insertion position
– Satisfaction
ACM Multimedia 2008 27
SatisfactionScore
Conventional Advertising
ImageSense
Ad position 3.00 3.75
Overall 3.68 3.85
<
<
Corbis images (http://pro.corbis.com/)
ACM Multimedia 2008 28
ACM Multimedia 2008 29
I II III IV V
Example - Tango
ACM Multimedia 2008 30
ACM Multimedia 2008 31
We
bs
ite
: Ta
ng
o
ACM Multimedia 2008 32
Live Search Images:“car”
Liv
e Im
ag
e S
ea
rch
Re
su
lts
ACM Multimedia 2008 33
1-3-1-1 1-3-1-2 1-3-1-3 1-3-1-4
1-3-1-6 1-3-1-7 1-3-1-8 1-3-1-9 1-3-1-10
1-3-1-11 1-3-1-13 1-3-1-14 1-3-1-15
1-3-1-16 1-3-1-19 1-3-1-20
Liv
e Im
ag
e S
ea
rch
Re
su
lts
VIPS Segmentation
ACM Multimedia 2008 34
Liv
e Im
ag
e S
ea
rch
Re
su
lts
Image Saliency Detection
ACM Multimedia 2008 35
Liv
e Im
ag
e S
ea
rch
Re
su
lts
Live Search Image with ImageSense
ACM Multimedia 2008 36
Website: Corbis
ACM Multimedia 2008 37
Conclusions
ImageSense support:
“contextually relevant” and “less-intrusive” advertising website / webpage / webimage
automate ad delivery by only inserting a piece of code
a long-tail business model(for both advertisers and publishers)
ACM Multimedia 2008 38
Thanks!
39 ACM Multimedia 2008
Thanks!
40 ACM Multimedia 2008
Implementation of ImageSense
ACM Multimedia 2008 42
(1) ImageSense service portal
(2) Publisher register a webpage
(3) ImageSenser provide a piece of html code
(4) Publisher’s page with ad