ranking and classifying attractiveness of photos in folksonomies jose san pedro and stefan...
TRANSCRIPT
Ranking and Classifying Ranking and Classifying Attractiveness of Attractiveness of Photos in FolksonomiesPhotos in Folksonomies
Jose San Pedro and Stefan SiersdorferJose San Pedro and Stefan Siersdorfer
University of Sheffield, L3S Research CenterUniversity of Sheffield, L3S Research Center
WWW 2009WWW 2009
22
OutlineOutline
IntroductionIntroduction
Features for Image AttractivenessFeatures for Image Attractiveness
Experiment DataExperiment Data
Experiment ResultsExperiment Results
ConclusionConclusion
33
OutlineOutline
IntroductionIntroduction
Features for Image AttractivenessFeatures for Image Attractiveness
Experiment SetupExperiment Setup
Experiment ResultsExperiment Results
ConclusionConclusion
44
IntroductionIntroduction
Thousands of new photos are uploaded to Thousands of new photos are uploaded to Flickr every minute.Flickr every minute.– Effective automatic content filtering is necessary.Effective automatic content filtering is necessary.
Meta data for Flickr photosMeta data for Flickr photos– TagsTags
– Number of viewsNumber of views
– User commentsUser comments
– Upload dateUpload date
– Save as favoriteSave as favorite
55
Attractive or Not?Attractive or Not?
Attractiveness of imagesAttractiveness of images– A highly subjective conceptA highly subjective concept
– Semantic aspects are associated but not crucialSemantic aspects are associated but not crucial People expression, picture composition… etcPeople expression, picture composition… etc
– The artistic component is an important factorThe artistic component is an important factor
– Low-level features are shown to provide high Low-level features are shown to provide high correlation with the attractivenesscorrelation with the attractiveness
66
ExampleExample
Figure: Attractive vs. Unattractive images. Each column Figure: Attractive vs. Unattractive images. Each column represents the same semantic concept (animal, landscape, represents the same semantic concept (animal, landscape, portrait, flower)portrait, flower)
77
OutlineOutline
IntroductionIntroduction
Features for Image AttractivenessFeatures for Image Attractiveness
Experiment SetupExperiment Setup
Experiment ResultsExperiment Results
ConclusionConclusion
88
Feature TypesFeature Types
Visual featuresVisual features– Color-associatedColor-associated
– Coarseness (sharpness)Coarseness (sharpness)
– Various color spaces are adoptedVarious color spaces are adopted RGB, YUV, HSL …etcRGB, YUV, HSL …etc
Text featuresText features– TagsTags
99
Color Features - 1Color Features - 1
Brightness Brightness – To measure the intensity of light waveTo measure the intensity of light wave
– YUV color spaceYUV color space
SaturationSaturation– To measure the vividnessTo measure the vividness
( Illustration of YUV color space, Y=0.5)
1010
Color Features - 2Color Features - 2
ColorfulnessColorfulness– To measure the difference against greyTo measure the difference against grey
1111
Color Features - 3Color Features - 3
Naturalness (Naturalness ( 自然度自然度 // 真實度真實度 ))– To measure the degree of correspondence between To measure the degree of correspondence between
images and human perception of realityimages and human perception of reality– HSL (Hue-Saturation-Lightness) color spaceHSL (Hue-Saturation-Lightness) color space– Pixels with 20 L 80 and S≦ ≦Pixels with 20 L 80 and S≦ ≦ >> 0.1 are grouped into 3 sets: 0.1 are grouped into 3 sets:
‘A – Skin’, ‘B – Grass’, ‘C – Sky’‘A – Skin’, ‘B – Grass’, ‘C – Sky’
1212
Color Features - 4Color Features - 4
ContrastContrast– To measure the relative variation of luminanceTo measure the relative variation of luminance
– RMS-contrastRMS-contrast
1313
Coarseness FeatureCoarseness Feature
Coarseness represents the degree of detail Coarseness represents the degree of detail contained in an image.contained in an image.
The most commonly used metric: The most commonly used metric: SharpnessSharpness– Be determined as a function of its Laplacian, normalized Be determined as a function of its Laplacian, normalized
by the local average luminanceby the local average luminance
where μwhere μxyxy denotes the average luminance around pixel (x,y) denotes the average luminance around pixel (x,y)
1414
Indication of TagsIndication of Tags
1515
OutlineOutline
IntroductionIntroduction
Features for Image AttractivenessFeatures for Image Attractiveness
Experiment SetupExperiment Setup
Experiment ResultsExperiment Results
ConclusionConclusion
1616
Experiment SetupExperiment Setup
Data from FlickrData from Flickr– 2.2M photos uploaded between June 1 and 7, 20072.2M photos uploaded between June 1 and 7, 2007– Among which 35,000 photos are with at least 2 favorite Among which 35,000 photos are with at least 2 favorite
assignmentsassignments– A random sample of 40,000 photos without any favorite A random sample of 40,000 photos without any favorite
assignment as the negative examplesassignment as the negative examples– The number of favorites are used as relevance valuesThe number of favorites are used as relevance values
Attractiveness classificationAttractiveness classification– Classifier: Support Vector Machine (SVMlight)Classifier: Support Vector Machine (SVMlight)
Ranking by attractivenessRanking by attractiveness– Regression model: Support Vector RegressionRegression model: Support Vector Regression
1717
OutlineOutline
IntroductionIntroduction
Features for Image AttractivenessFeatures for Image Attractiveness
Experiment SetupExperiment Setup
Experiment ResultsExperiment Results
ConclusionConclusion
1818
Classification Results - Classification Results - 11 Table: Classification results (BEP) of 500 Table: Classification results (BEP) of 500
“attractive/unattractive” training photos“attractive/unattractive” training photos
1919
Classification Results - Classification Results - 22 Table: Classification results (BEP) of 8000 Table: Classification results (BEP) of 8000
“attractive/unattractive” training photos“attractive/unattractive” training photos
2020
Classification Results - Classification Results - 33
Figure: Precision-recall curves for visual and textual Figure: Precision-recall curves for visual and textual dimensions and their combination (8000 training photos per dimensions and their combination (8000 training photos per class, numFav 5)≧class, numFav 5)≧
2121
Ranking ResultRanking Result
Evaluated by Kendall’s Tau-bEvaluated by Kendall’s Tau-b
Table: Ranking using regressionTable: Ranking using regression
2222
OutlineOutline
IntroductionIntroduction
Features for Image AttractivenessFeatures for Image Attractiveness
Experiment SetupExperiment Setup
Experiment ResultsExperiment Results
ConclusionsConclusions
2323
ConclusionsConclusions
We have used favorite assignments in Flickr to We have used favorite assignments in Flickr to obtain training data for attractiveness obtain training data for attractiveness classification and ranking.classification and ranking.
The best performance is achieved by The best performance is achieved by combining tags and visual information.combining tags and visual information.
We plan to extend and generalize this work to We plan to extend and generalize this work to consider various kinds of resources such as consider various kinds of resources such as videos or text.videos or text.
2424