acm mm 2010 - ee.columbia.edudongliu/slides/imageretagging.pdf · cat kitty cat feline vertebrate...
TRANSCRIPT
![Page 1: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/1.jpg)
Dong Liu, Xian-Sheng Hua, Meng Wang and Hong-Jiang Zhang
Harbin Institute of Technology
Microsoft Research Asia
Microsoft Advanced Technology Center
ACM MM 2010
![Page 2: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/2.jpg)
2
medici chapel, Firenze, Italy... Loggia dei lanzi, sword, honeymoon, ...
Status, building, sky, Italy, ... Cathedral, tower, Italy...
![Page 3: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/3.jpg)
Social tags are good, but they are
Noisy
Ambiguous
Incomplete
No relevance information
Two directions to improve tag quality
Tag Ranking (Liu, Hua, Zhang. Tag Ranking. WWW 09)
Retagging (Liu, Hua, Wang, Zhang. Image Retagging. MM 10)
3
![Page 4: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/4.jpg)
kitty
boy
top101
young
lovely
Imprecise Tags
Subjective Tags
Missing Tags
Tags associated with social images are imprecise, subjective and incomplete.
grass flower cat animal 4
![Page 5: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/5.jpg)
5
To improve:
Tag-based image search
Image annotation (automatic tagging)
What we are going to do:
Improve the quality of the tags to better describe content.
dog
house
tree
sky
ground
cloud
top 101
tour
tiger
sweet
big
cloud
![Page 6: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/6.jpg)
6
But how can we make it?
Automatically.
![Page 7: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/7.jpg)
Similar images similar tags
User-provided tags correlate with the image content with high probability
7
bear
water
wildlife
nature
river
bear
animal
bath
power
boy
zoo
tiger
father
Nikon
cat
animal
garden
rabit
Visual and Semantic Consistency
Prior Knowledge
![Page 8: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/8.jpg)
Tag Refinement The consistency between visual similarity and semantic similarity should be maximized.
The deviation from the initially user-provided tags should be minimized.
8
![Page 9: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/9.jpg)
9
Notations
![Page 10: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/10.jpg)
Visual and semantic consistency
User-provided tags are relevant with high probability
10
Modeling the basic assumptions
Overall formulation
![Page 11: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/11.jpg)
Optimizing with iterative updating
11
Bound the objective function
Derive the solution
Iterative updating until convergence
![Page 12: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/12.jpg)
12
Is It Reliable ?
![Page 13: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/13.jpg)
13
bird
Content -Related Tag Content -Unrelated Tag
Describe the REAL visual content of the images. Informative for ALL general users.
Describe the CONTEXTUAL information about the images . Only informative to the image owners.
beach
bike sunset
flower animal
baby
autumn
night
dog
ocean cat grass fun
deleteme top101
best Nikon
photo
old
my
macro
raw
science
live
![Page 14: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/14.jpg)
14
Similar images have similar tags.
Involving the content-unrelated tags will
Introduce lots of noises.
Degrade the algorithmic performance.
Only applicable for “content-related” tags.
RECALL
These tags should be removed from the automatic learning procedure.
![Page 15: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/15.jpg)
Filter out all content-unrelated tags.
15
All words
Non-noun
Noun
content-unrelated
content-related
Organism
Natural Pho.
Thing
Artifact
Color
kitty
feline
mammal
animal
organism
building
structure
artifact
Construct a content-related tag dictionary by using the lexical and domain knowledge
Traverse along the path until one pre-defined category is matched
![Page 16: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/16.jpg)
16
Is It Enough?
![Page 17: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/17.jpg)
17
kitty kitty
kitten cat
pussy
synonym
feline animal
hypernym
kitty
The missing of such tags will degrade the performance of tag-based applications.
![Page 18: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/18.jpg)
Make use of Wordnet lexicon
18
domestic
cat chordate organism kitty cat animal feline vertebrate
kitty-cat kitty kitten pussy pussycat
Use each tag to perform tag-based image search on Flickr.
The tags with more than 10,000 returned images are retained.
![Page 19: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/19.jpg)
19
Three-step strategy
![Page 20: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/20.jpg)
In term of average precision,recall and F1-Measure 50,000 Flickr images with 4,556 content-related tags.
2,500 test images.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Precision Recall F1-Measure
Original CBAR Our Method
20
![Page 21: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/21.jpg)
![Page 22: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/22.jpg)
22
Method Precision Recall F1-measure Relevant tag num
Before Enrichment
0.71 0.34 0.46 3.09 (4,80 in all )
After Enrichment
0.90 0.66 0.76 9.34 (10.38 in all)
Tagging quality is further improved rafter the tag enrichment procedure.
![Page 23: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/23.jpg)
Use the learnt confidence scores as relevance measure
Ranking results for query “cat”
23
![Page 24: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/24.jpg)
24
Our confidence score based ranking strategy outperforms the other image ranking strategies on Flickr
![Page 25: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/25.jpg)
Use top tags of the images after retagging to predict the tags of the unlabeled images
25
![Page 26: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/26.jpg)
26
Using top tags after image retagging can obtain better results than using the original images directly
![Page 27: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/27.jpg)
27
User-provided tags are imprecise and incomplete, which limits the performance of tag-based applications
We propose an image retagging strategy to solve this problem: Tag filtering to remove the content-unrelated tags
Tag refinement to automatically refine the tags
Tag enrichment to expand the tags with synonyms and hypernyms.
Image retagging benefits a series of tag-based applications
![Page 28: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/28.jpg)
Extend it to online videos
Using more fruitful information cues such as image regions and surrounding texts
28
![Page 29: ACM MM 2010 - ee.columbia.edudongliu/Slides/ImageRetagging.pdf · cat kitty cat feline vertebrate chordate animal organism kitty kitty-cat kitten pussy pussycat Use each tag to perform](https://reader034.vdocuments.site/reader034/viewer/2022043010/5fa2952dfb8d2f5b3471cfb9/html5/thumbnails/29.jpg)
29