ranking and classifying attractiveness of photos in folksonomies jose san pedro and stefan...

Ranking and Classifying Ranking and Classifying Attractiveness of Attractiveness of Photos in FolksonomiesPhotos in Folksonomies

Jose San Pedro and Stefan SiersdorferJose San Pedro and Stefan Siersdorfer

University of Sheffield, L3S Research CenterUniversity of Sheffield, L3S Research Center

WWW 2009WWW 2009

22

OutlineOutline

IntroductionIntroduction

Features for Image AttractivenessFeatures for Image Attractiveness

Experiment DataExperiment Data

Experiment ResultsExperiment Results

ConclusionConclusion

33

OutlineOutline

IntroductionIntroduction

Features for Image AttractivenessFeatures for Image Attractiveness

Experiment SetupExperiment Setup

Experiment ResultsExperiment Results

ConclusionConclusion

44

IntroductionIntroduction

Thousands of new photos are uploaded to Thousands of new photos are uploaded to Flickr every minute.Flickr every minute.– Effective automatic content filtering is necessary.Effective automatic content filtering is necessary.

Meta data for Flickr photosMeta data for Flickr photos– TagsTags

– Number of viewsNumber of views

– User commentsUser comments

– Upload dateUpload date

– Save as favoriteSave as favorite

55

Attractive or Not?Attractive or Not?

Attractiveness of imagesAttractiveness of images– A highly subjective conceptA highly subjective concept

– Semantic aspects are associated but not crucialSemantic aspects are associated but not crucial People expression, picture composition… etcPeople expression, picture composition… etc

– The artistic component is an important factorThe artistic component is an important factor

– Low-level features are shown to provide high Low-level features are shown to provide high correlation with the attractivenesscorrelation with the attractiveness

66

ExampleExample

Figure: Attractive vs. Unattractive images. Each column Figure: Attractive vs. Unattractive images. Each column represents the same semantic concept (animal, landscape, represents the same semantic concept (animal, landscape, portrait, flower)portrait, flower)

77

OutlineOutline

IntroductionIntroduction

Features for Image AttractivenessFeatures for Image Attractiveness

Experiment SetupExperiment Setup

Experiment ResultsExperiment Results

ConclusionConclusion

88

Feature TypesFeature Types

Visual featuresVisual features– Color-associatedColor-associated

– Coarseness (sharpness)Coarseness (sharpness)

– Various color spaces are adoptedVarious color spaces are adopted RGB, YUV, HSL …etcRGB, YUV, HSL …etc

Text featuresText features– TagsTags

99

Color Features - 1Color Features - 1

Brightness Brightness – To measure the intensity of light waveTo measure the intensity of light wave

– YUV color spaceYUV color space

SaturationSaturation– To measure the vividnessTo measure the vividness

( Illustration of YUV color space, Y=0.5)

1010

Color Features - 2Color Features - 2

ColorfulnessColorfulness– To measure the difference against greyTo measure the difference against grey

1111

Color Features - 3Color Features - 3

Naturalness (Naturalness ( 自然度自然度 // 真實度真實度 ))– To measure the degree of correspondence between To measure the degree of correspondence between

images and human perception of realityimages and human perception of reality– HSL (Hue-Saturation-Lightness) color spaceHSL (Hue-Saturation-Lightness) color space– Pixels with 20 L 80 and S≦ ≦Pixels with 20 L 80 and S≦ ≦ ＞＞ 0.1 are grouped into 3 sets: 0.1 are grouped into 3 sets:

‘A – Skin’, ‘B – Grass’, ‘C – Sky’‘A – Skin’, ‘B – Grass’, ‘C – Sky’

1212

Color Features - 4Color Features - 4

ContrastContrast– To measure the relative variation of luminanceTo measure the relative variation of luminance

– RMS-contrastRMS-contrast

1313

Coarseness FeatureCoarseness Feature

Coarseness represents the degree of detail Coarseness represents the degree of detail contained in an image.contained in an image.

The most commonly used metric: The most commonly used metric: SharpnessSharpness– Be determined as a function of its Laplacian, normalized Be determined as a function of its Laplacian, normalized

by the local average luminanceby the local average luminance

where μwhere μxyxy denotes the average luminance around pixel (x,y) denotes the average luminance around pixel (x,y)

1414

Indication of TagsIndication of Tags

1515

OutlineOutline

IntroductionIntroduction

Features for Image AttractivenessFeatures for Image Attractiveness

Experiment SetupExperiment Setup

Experiment ResultsExperiment Results

ConclusionConclusion

1616

Experiment SetupExperiment Setup

Data from FlickrData from Flickr– 2.2M photos uploaded between June 1 and 7, 20072.2M photos uploaded between June 1 and 7, 2007– Among which 35,000 photos are with at least 2 favorite Among which 35,000 photos are with at least 2 favorite

assignmentsassignments– A random sample of 40,000 photos without any favorite A random sample of 40,000 photos without any favorite

assignment as the negative examplesassignment as the negative examples– The number of favorites are used as relevance valuesThe number of favorites are used as relevance values

Attractiveness classificationAttractiveness classification– Classifier: Support Vector Machine (SVMlight)Classifier: Support Vector Machine (SVMlight)

Ranking by attractivenessRanking by attractiveness– Regression model: Support Vector RegressionRegression model: Support Vector Regression

1717

OutlineOutline

IntroductionIntroduction

Features for Image AttractivenessFeatures for Image Attractiveness

Experiment SetupExperiment Setup

Experiment ResultsExperiment Results

ConclusionConclusion

1818

Classification Results - Classification Results - 11 Table: Classification results (BEP) of 500 Table: Classification results (BEP) of 500

“attractive/unattractive” training photos“attractive/unattractive” training photos

1919

Classification Results - Classification Results - 22 Table: Classification results (BEP) of 8000 Table: Classification results (BEP) of 8000

“attractive/unattractive” training photos“attractive/unattractive” training photos

2020

Classification Results - Classification Results - 33

Figure: Precision-recall curves for visual and textual Figure: Precision-recall curves for visual and textual dimensions and their combination (8000 training photos per dimensions and their combination (8000 training photos per class, numFav 5)≧class, numFav 5)≧

2121

Ranking ResultRanking Result

Evaluated by Kendall’s Tau-bEvaluated by Kendall’s Tau-b

Table: Ranking using regressionTable: Ranking using regression

2222

OutlineOutline

IntroductionIntroduction

Features for Image AttractivenessFeatures for Image Attractiveness

Experiment SetupExperiment Setup

Experiment ResultsExperiment Results

ConclusionsConclusions

2323

ConclusionsConclusions

We have used favorite assignments in Flickr to We have used favorite assignments in Flickr to obtain training data for attractiveness obtain training data for attractiveness classification and ranking.classification and ranking.

The best performance is achieved by The best performance is achieved by combining tags and visual information.combining tags and visual information.

We plan to extend and generalize this work to We plan to extend and generalize this work to consider various kinds of resources such as consider various kinds of resources such as videos or text.videos or text.

2424