Exploiting Flickr Tags and Groups for Finding Landmark

Photos

short paper at ECIR 2009

Rabeeh Abbasi, Sergey Chernov, Wolfgang Nejdl, Raluca Paiu, and Steffen Staab

{abbasi,staab}@uni-koblenz.de, {chernov,nejdl,paiu}@L3S.de

Intro: Landmarks vs Non-Landmarks Problem

Tag “Beijing”

Finding Landmark Photos

Goal Develop a method for easy classification of resources

Idea Exploit Flickr Groups (http://www.flickr.com/groups)

Method Select groups related to positive

and negative classes for training examples

Create and normalize feature space Train the classifier Classify unknown images

Applications Helps in improving search and

browsing of resources related toparticular class(es)

Query

flickr.com/photos/swamibu/2223726960/, flickr.com/photos/gunner66/2219882643/, flickr.com/photos/mromega/2346732045/, flickr.com/photos/me_haridas/399049455/, flickr.com/photos/caribb/84655830/, flickr.com/photos/conner395/1018557535/, flickr.com/photos/66164549@N00/2508246015/, flickr.com/photos/kupkup/1356487670/, flickr.com/photos/asam/432194779/, flickr.com/photos/michaelfoleyphotography/392504158/

ClassifierClassifier

Norm

alization

ClassificationModelSVMSVM

+VE -VE

PositiveTraining

Examples

NegativeTraining

Examples

Positive Flickr Groups

Negative Flickr Groups

Problem decomposition

2. Classify all photos into landmarks and non-

landmarks

1. Select all photos containing city tag

3. Rank tags from landmark photos by

likelihood to represent a landmark

4. Present top-k photos containing most

prominent landmark tags

Tag Normalization for ClassificationU - users, T - tags, R - resources (photos)Normalized tag frequency of a tag t in a resource rTag Frequency:

TFr(t) = t tag counts per photo / total tags per photoInversed Resource Frequency:

IRF(t) = log (total number of photos / number of photos having t)Inversed User Frequency:

IUF(t) = log (total number of users / number of user having t)

Feature vector for a photo r:F(r) = [TFr(t1)*IRF(t1); TFr(t2)*IRF(t2); … ; TFr(tjTj) IRF(tjTj)]

Experiments with Normalization Schema

Best schema:

F(r) = TFr(t)*IRF(t)

Measuring Tag Representativeness

City Tag Frequency, City User Tag Frequency, Confindence:

t tag counts across landmark photos for a city

CTF(t) = maximum tag counts across landmark photos for a city

number of users having tag t across landmark photos for a city

CUTF(t) = maximum number of users having tag t across landmark photos for a city

CONF(t) = log (sum of confidence scores produced by SVM from all photos with t)

RepresentativenessScore(t) = IRF(t) * IUF * CTF(t) * CUTF(t) * CONF(t)

Evaluation

Experimental Setup

Datasets - Training Dataset (430k photos) - Test Dataset (232k photos)

Comparison with state-of-art system WorldExplorer (Yahoo!)

20 Users evaluated both methods for 50 cities

Evaluation Prototype

The users were asked to judge if a photo is a landmark or not

Around 400-500 judgmentsper user (30 minutes per user).

20 users, each user evaluated two result sets (mixed together) for 10 randomly selected cities out of 50, each city is evaluated by 4 users in total

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

Precision per UserWorld Explorer Our Method

User

Prec

ision

Micro-average Precision: World Explorer = 0.33 Our method = 0.37

Statistically significant

dres

den

mos

cow

ista

nbul

cairo

liver

pool

bras

iliaro

me

shan

ghai

flore

nce

prag

uetu

rinro

tterd

amco

logn

ebe

rlin

sevi

lleba

rcel

ona

geno

aam

ster

dam

glas

gow

napl

espa

lerm

oha

mbu

rgbu

char

est

hann

over

mad

ridhe

lsin

kivi

enna

beijin

gat

hens

chic

ago

birm

ingh

ambu

enos

aire

ssa

ntia

gone

wyo

rkm

unic

hto

ront

ole

eds

hong

kong

riode

jane

iroto

kyo

oslo

losa

ngel

espa

rislo

ndon

stoc

khol

myo

koha

ma

sing

apor

esa

opau

lom

exic

ocity

sydn

ey

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Precision per City

World Explorer Our Method

Pre

cisi

on

Macro-average Precision: World Explorer = 0.33 Our method = 0.34

Not statistically significant, variance is too high

Conclusions

1. Precision improvement of 12%, (80% users preferred our method 60% cities are better than WE with our method

2. Landmark finding based on photo classification can replace geo-tagging based methods in situations where geo-spatial information is not available

The algorithm has a potential to be generalized beyond city landmarks for any topical photos, such as “cars", “mobile phones“, etc.

Thank You!