Finding Celebrities in Billions of Web Images

云飞2012-12-13

• 一、 label an input image with a list of celebrities.

• 二、 the celebrity names are assigned to the faces by label propagation on a facial similarity graph.

Overview

• 本文的优点：• 1 、 the proposed image annotation system is

capable of labeling names to general web images.

• 2 、 our name assignment algorithm does not impose any assumption on the facial feature distribution.

• 3 、 not only visual cues are used.

Overview

• 1. determine, by identifying celebrity names from surrounding text.

• 2. given a set of names, assign the names to the faces in the input image.

Overview

• A. Image Annotation System• 1) construct a vocabulary;• 2) discover all webpages hosting its near-duplicates;• 3) use the vocabulary to filter the surrounding text.

– Advances：• 1)effective;• 2)remove noise.

– Annotated images:• 1)SFSN• 2)SFMN• 3)MF

Overview

• B. Multimodal Name Assignment• The context likelihood incorporates the information

from surrounding text by using the confidence scores estimated by the image annotation system.

Overview

IMAGE ANNOTATION SYSTEM

• Goal: label an input image with a list of celebrities who may appear in the image.

• A. Constructing a Large-Scale Celebrity Name Vocabulary• B. Discover Related Webpages by Near-Duplicate Image

Retrieval• C. Annotating Images by Mining Surrounding Text of

Related Webpages

IMAGE ANNOTATION SYSTEM

A. Constructing a Large-Scale Celebrity Name Vocabulary1)Wikipedia首段信息框标签

2)Entitycube

IMAGE ANNOTATION SYSTEM

• B. Discover Related Webpages by Near-Duplicate Image Retrieval– divide and conquer strategy• 图片分成 n×n• 降维• 阈值化

IMAGE ANNOTATION SYSTEM

• C. Annotating Images by Mining Surrounding Text of Related Webpages

• 1) Type of names；• 2) Type of surrounding text；• 3) Frequency versus ratio；

• A. Notation• B. Overview of the Assignment Model• C. Label Propagation from SFSN Images p(Y|F)• D. Constrain the Propagation by a Context

Likelihood p(Y|T; λ)• E. Normalization by Name Prior p(Y)• F. Implementation Detail: Face Representation

MULTIMODAL NAME ASSIGNMENT

• A. Notation– faces in image In

– denote the face labels as

• B. Overview of the Assignment Model– the confidence for label

• C. Label Propagation from SFSN Images p(Y|F)– how to propagate labels from SFSN images to

SFMN and MF images

• D. Constrain the Propagation by a Context Likelihood p(Y|T; λ)

• 1) For each image-level name vk, generate a binary variable zk from p(vk |T) as defined in (3) to indicate whether vk appears in image I.

• 2) If zk=1, generate mk faces of name vk in image I from p(m|z; λ) as defined in (13).

• E. Normalization by Name Prior p(Y)– p(Y) represents the prior of names.

• F. Implementation Detail: Face Representation• the appearance of each face is described by local binary

pattern (LBP).• the face image is divided into small regions from which

the LBP features are extracted and concatenated into a single feature histogram.• pply PCA to reduce the dimension of face descriptor

from over 3000 to 500 dimensions.

Evaluation