sewm'14 keynote: mining events from multimedia streams

M I N I N G E V E N T S F R O M M U LT I M E D I A S T R E A M S

J O N A T H O N H A R E A N D S I N A S A M A N G O O E I

A P P L E W W D C K E Y N O T E 2 0 1 3

“Life is full of special moments. So is your photo library.”

B U T W H AT I F W E G O B E Y O N D P E R S O N A L M E D I A L I B R A R I E S ?

C A N W E A U T O M AT I C A L LY F I N D M E A N I N G F U L E V E N T S A N D T R E N D S I N S T R E A M S O F S O C I A L M U LT I M E D I A ?

C O N T E N T S

• Research challenges

• Case studies using shared social media:

• Monitoring Twitter’s visual pulse

• Detecting social events

• Open questions & future directions

R E S E A R C H C H A L L E N G E S 1

• How do we deal with massive amounts of data?

• There is a temporal aspect to the data we’re dealing with…

• Develop streaming algorithms that can be:

• Incrementally updated

• Allowed to forget

R E S E A R C H C H A L L E N G E S 2

• How can we effectively make use of contextual data and metadata from different modalities?

• Develop techniques for exploiting all modalities and fusing features from each modality effectively

• Develop techniques that are robust to missing or inaccurate features

M O N I T O R I N G T W I T T E R ’ S V I S U A L P U L S E

P H O T O S H A R I N G ( B R O A D C A S T I N G ? ) O N T W I T T E R

• Photos are heavily shared on Twitter

• In 2012 we were capturing 100,000 images per day from the sample stream

• Equates to 70GB of data

• Sample stream is supposedly ~1% of all tweets

• Implying on the firehose upwards of 10,000,000 images per day

• Thats 7TB of data per day

W H AT T Y P E S O F P H O T O S A R E S H A R E D O N T W I T T E R ?

• Small study performed in September 2012

• Aim to get a feel for what is being Tweeted

• ~1 Day’s worth of data chosen (81368 images)

• 14 trusted Human annotators from our research group

• Uniform random sample of 667 images annotated

0"

50"

100"

150"

200"

250"

300"

350"

400"

450"

anim

al:bird

"anim

al:bird

:several"

anim

al:dinsour"

anim

al:ki7en

s"anim

al:ra

bbit"

color:b

lue"

object:glasses"

object:m

oney"

object:spe

ctacles"

object:watch"

scen

e:ou

tdoo

rs:nature"

sen@

men

t:sad"

sport"

text:dutch"

text:german"

text:italian"

text:portuguese"

@me:night"

anim

al:fish"

art:sculpture"

Baby"

object:jewellery"

object:m

otorbike"

object:new

spaper"

object:vegetable"

scen

e:concert"

text:cyrillic"

type

:wD"

art:s@llElife

"ob

ject:album

Ecover"

object:sign"

type

:awesom

e"type

:manga"

art:d

rawing"

art:p

ain@

ng"

object:fu

rnitu

re"

type

:advert"

anim

al:dog"

art:fi

lmEfram

es"

color:b

lackEand

Ewhite"

object:car"

object:ornam

ent"

object:to

y"sen@

men

t:disturbing"

object:boo

k"ob

ject:veh

icle"

ac@vity

:music"

object:fa

shion"

peop

le:porn"

type

:promoEpo

ster"

object:building"

object:drin

k"ob

ject:sho

es"

type

:mem

e"ob

ject:clothes"

sen@

men

t:hum

orou

s"ac@vity

:spo

rt"

text:spanish"

peop

le:child"

peop

le:children"

peop

le:par@a

llyEnaked

"ob

ject:te

chno

logy"

peop

le:partEofEhum

an"

anim

al:cat"

art:p

oster"

text:arabic"

peop

le:crowd"

scen

e:ou

tdoo

r:sub

urban"

type

:screensho

t"scen

e:ou

tdoo

r:landscape

"pe

ople:group

Epose"

scen

e:ou

tdoo

rs:city

"type

:cartoon

"pe

ople:m

ugEsho

t"ob

ject:fo

od"

text:unkno

wn"

type

:collage"

scen

e:indo

or"

type

:graph

ic"

peop

le:several"

text:english"

peop

le:in

dividu

al"

type

:pho

tograph"

1 D AY ’ S W O R T H O F D ATA C H O S E N !1 4 A N N O TAT O R S !

0"

50"

100"

150"

200"

250"

300"

350"

400"

450"

type:advert"

type:awesome"

type:cartoon"

type:collage"

type:graphic"

type:m

anga"

type:m

eme"

type:photograph"

type:prom

o;poster"

type:screenshot"

type:w

<"

TYPES

0"

10"

20"

30"

40"

50"

60"

70"

80"

90"

text:arabic"

text:cyrillic"

text:dutch"

text:english"

text:germ

an"

text:italian"

text:portuguese"

text:spanish"

text:unknown"

TEXT

0"

20"

40"

60"

80"

100"

120"

140"

160"

people:child"

people:children"

people:crowd"

people:group6pose"

people:individual"

people:mug6shot"

people:part6of6human"

people:par=ally6naked"

people:porn"

people:several"

PEOPLE

0"

2"

4"

6"

8"

10"

12"

14"

16"

18"

animal:bird"

animal:bird:several"

animal:cat"

animal:dinsour"

animal:dog"

animal:fish"

animal:ki<ens"

animal:rabbit"

ANIMALS

0"

5"

10"

15"

20"

25"

30"

35"

40"

object:album3cover"

object:book"

object:building"

object:car"

object:clothes"

object:drink"

object:fashion"

object:food"

object:furniture"

object:glasses"

object:jewellery"

object:money"

object:motorbike"

object:newspaper"

object:ornament"

object:shoes"

object:sign"

object:spectacles"

object:technology"

object:toy"

object:vegetable"

object:vehicle"

object:watch"

OBJECTS

C A N W E F I N D W H AT I M A G E S A R E T R E N D I N G O N T W I T T E R ?

• Look for near duplicates of images we’ve seen before • can’t just use the URL…

• the same image appears at many different URLs • potentially with slight changes

• compression • added text • scaling • rotation/warps

• Don’t need to consider all images we’ve seen in the past… • maybe just those in a sliding time window

• Need a fast and robust near duplicate detection technique that can be adapted to working with a stream.

• Want to use a local feature (i.e. DoG-SIFT) for robustness

• BoVW+Inverted index, VLAD+PQ not ideal

• …use locality sensitive hashing…

• Idea inspired by method of Dong et al at ICMR’12

PicSlurperFeature

Extractor

Random LSHs

HashTables

1 2 3 4

BitString Partitioner

images

features

LSH sketches

Duplicate Graph Construction

Connected Component Labelling

Duplicates

Pruning of Old Images

W O R K F L O W

HashTable

1

212

...

image1, image4

image2, image54, ...

...

783 image4, image4096image1 image4

image54image2

image4096

4 HASH TABLES

32-BIT KEY

VALUES ARE LISTS OF IMAGES

ANY COLLISION RESULTS IN A GRAPH EDGE

CONNECTED COMPONENTS DUPLICATES

D E M O

S O C I A L E V E N T D E T E C T I O N

M E D I A E VA L 2 0 1 3 : S E D TA S K

• Open evaluation challenge at MediaEval 2013: • http://www.multimediaeval.org/ • Event clustering of multimodal social streams

!

• Specifically: • Detect social events…

• Given 500k flickr images with • image, tags, (some) geo, (some) time take, time

posted etc.

http://www.multimediaeval.org/

http://www.multimediaeval.org/

W H AT A R E S O C I A L E V E N T S ?

We define social events as events that are planned by people, attended by people and the

media illustrating the events are captured by people

S E D 2 0 1 3 E X A M P L E S

< P H O T O I D = " 4 3 0 2 7 4 6 4 2 9 " P H O T O _ U R L = " H T T P : / /FA R M 5 . S TAT I C F L I C K R . C O M /4 0 6 8 / 4 3 0 2 7 4 6 4 2 9 _ F 8 C D 7 F 2 5 8 2 . J P G " U S E R N A M E = " AT T H E F O O T O F T H E H I L L " D AT E TA K E N = " 2 0 1 0 - 0 1 - 2 3 2 1 : 3 6 : 0 5 . 0 " D AT E U P L O A D E D = " 2 0 1 0 - 0 1 - 2 5 1 0 : 4 3 : 3 4 . 0 " > < T I T L E > P O O , D E A D V O I C E S O N A I R @ A 4 < / T I T L E > < D E S C R I P T I O N > W H O L E S E T . . . < / D E S C R I P T I O N > < TA G S > < TA G > 2 / 5 8 < / TA G > < TA G > 5 8 M M < / TA G > < TA G > C O N C E R T < / TA G > < TA G > F 2 < / TA G > < TA G > P O O < / TA G > < / TA G S > < / P H O T O >

< P H O T O I D = " 2 2 8 0 0 6 0 8 5 2 " P H O T O _ U R L = " H T T P : / /FA R M 3 . S TAT I C F L I C K R . C O M /2 1 1 0 / 2 2 8 0 0 6 0 8 5 2 _ 1 6 5 3 9 F 5 F 0 D . J P G " U S E R N A M E = " P I L O T _ 1 0 " D AT E TA K E N = " 2 0 0 8 - 0 2 - 1 9 2 3 : 3 4 : 0 9 . 0 " D AT E U P L O A D E D = " 2 0 0 8 - 0 2 - 2 0 1 8 : 3 9 : 5 0 . 0 " > < T I T L E > J E N S L E K M A N @ T E AT R O R A S I , R AV E N N A < /T I T L E > < D E S C R I P T I O N > 1 9 F E B B R A I O 2 0 0 8 < / D E S C R I P T I O N > < TA G S > < TA G > A L L R I G H T S R E S E R V E D < / TA G > < TA G > C O N C E R T I < / TA G > < TA G > C O O L P I X _ 4 3 0 0 < / TA G > < TA G > E M I L I A R O M A G N A < / TA G > < / TA G S > < L O C AT I O N L AT I T U D E = " 4 4 . 4 1 5 3 " L O N G I T U D E = " 1 2 . 2 0 5 2 " > < / L O C AT I O N > < / P H O T O >

B U I L D S PA R S E M A T R I X

Sparse Feature Matrix

CombinedSparseMatrix




B U I L D S PA R S E M A T R I X


CombinedSparseMatrix




DBSCAN

Spectral

+ Incremental

C L U S T E R

F E AT U R E S

• Events potentially separable using: • Images: should look similar? • Time: should be temporally close? • Location: should be geographically close? • Text: should be described similarly?

• Flickr multimodal social stream contains: • Time taken (potentially inaccurate or missing entirely) • Time posted (accurate, though may be event agnostic) • Geo (often inaccurate, sparse) • Tags, title, description (multi-tag, spelling etc.) • Image features (these didn’t work so well!)

H T T P S : / / G I T H U B . C O M / S I N J A X / S O T O N - M E D I A E VA L 2 0 1 3

https://github.com/sinjax/soton-mediaeval2013

E V E N T S E PA R AT I O N W I T H F E AT U R E S

• January 2007 until February 2007

• Random color assigned to clusters


E V E N T S L I K E T H I S C O U L D E A S I LY B E C O N F U S E D


M O R E F E A T U R E S M A Y H E L P S E PA R A T E E V E N T S

F E AT U R E W E I G H T S

• The features matter for different reasons

• Some are more important than others

• This is a feature fusion problem!

• We went with a simple strategy: weighted summation

• Use simplex search to find what feature weights give the best cluster quality

F E AT U R E W E I G H T S I M P L E X

F E AT U R E W E I G H T S

• Time taken is apparently most important

• Time posted + geo seem to hold the same information

• Tags beat titles and descriptions

C L U S T E R I N G – F I N D I N G E V E N T S

• Clustering specific challenges • Number of clusters hard to estimate

• But, it’s a parameter in many clustering algorithms • Many noise points

• 2% clusters with 1 member • Clustering a stream

• The size of the streams • Updating clusters

C L U S T E R I N G – D B S C A N

• DBSCAN is an old, well studied clustering algorithm

• Detects clusters and identify noise

• No knowledge of cluster count needed

• Requirements: • Neighbourhood function

(e.g. thresholded sparse similarity matrix)

• Neighbourhood density counts

• Find mutually density-connected items

H T T P : / / B I T. LY / O I C L U S T E R I N G

http://bit.ly/oiclustering


C L U S T E R I N G - S P E C T R A L

• Theoretically appealing non-parametric clustering algorithm • Rooted in graph theory • Potentially auto detects

cluster count • Basic premise is:

• Use the smallest (near zero) eigenvalued eigenvectors of the graph laplacian of the similarity matrix of some data as a space within which to apply another clustering algorithm




C L U S T E R I N G - I N C R E M E N TA L

• Practical restrictions of spectral clustering mean we can’t apply it to the whole dataset

• Make an assumption about the data style

• Images likely to be clustered together will appear sequentially in terms of upload time

• Leverage this to cluster sub windows of data


Cluster Initial Window

1) Grow window, cluster again

2)

...

Identify stable clusters

3) Continue, ignoring stable clusters

4)



M E D I A E VA L S E D TA S K R E S U LT S

• Calibration on 300k training items to optimise: • Feature weightings • Clustering parameters

!

• Clustering performed on a 200k item test set • In the 2013 SED task, this technique:

• produced the best F1 scores; • was one of the only streaming approaches; • was one of the few open-source approaches

R E S U LT S

6

TABLE II: Results from MediaEval 2013. Our 4 submittedruns are presented last and in bold. The best results over allruns are also highlighted.

Group/Technique F1 NMI divF1

CERTH-ITI(1) [13] 0.8865 0.8739 0.5701CERTH-ITI(2) [14] 0.7031 0.9131 0.6367UPC [15] 0.8798 0.9720 0.8268UNITN [16] 0.9320 0.9849 0.8793TUWIEN [17] - 0.94 0.78ADMRG [18] 0.812 0.954 0.758ISMLL [19] 0.8755 0.9641 -NTUA [20] 0.2364 0.6644 -VIT [21] 0.1426 0.1802 0.0724QM [22] - 0.94 0.78DBSCAN (best-weight) 0.9454 0.9851 0.8865Spectral (best-weight) 0.9114 0.9765 0.8534DBSCAN (average-weight) 0.9461 0.9852 0.8864Spectral (average-weight) 0.9024 0.9737 0.8455

of the relative methods of other participants who came closeto our state of the art results.

V. RELATED WORK

As outlined by Scherp et al. [23] there is a great deal ofinterest in detecting multimedia related to high level occur-rences in which humans participate. A subtype of such eventsare the Social Events the MediaEval Social Event Detection(SED) task asks participants to detect. The challenge distin-guishes Social Events as those events which were “plannedby people, attended by people and that social media depictingthe events are taken by people”. This calls for events beyondsuch definitions as birthday party and towards more specificdefinitions such as Sina’s 30th Birthday Party.

When attempting to organise multimedia items into thosebelonging to the same social events temporal and spatial meta-data are the most powerful indicators of event membership. Itis clear that if the true time and true location of a multimediaitem could ever be known with complete accuracy, and ifan assumption is made that all the multimedia items beingprocessed represent events, then the clustering of items intosocial events would be made far easier in most contexts. This isstated explicitly in the problem definition provided by Beckeret al. [24] who say that an event is something that: “... occursin a certain place at a certain time...” Indeed, it is difficultto imagine two items of multimedia taken in the same placeat the same time which would not, in some sense, depict thesame social event. A strong edge case of this statement is arestaurant which might have many, though separate, groups ofpeople of an evening. Though the restaurant is one physicallocation, it is reasonable to assume that each table in therestaurant might be host to different social events. Thereforetwo multimedia items from two different groups present,though geographically and temporally similar, should not beassigned to a single social event. This edge case highlightsthe issue of scale. Namely, if an event occurs across a largeenough such that sub geographic locations could feasiblyhold unrelated events, then even precise geographic and timeinformation alone cannot help us distinguish events. This issuenotwithstanding, precise geographic and time information canachieve most of the desired goal in a standard SED task. Many

of the approaches to social event detection highlight timeand geographic location as the most important dimensions ofseparation in their solutions. A majority of their other effortseither explicitly or implicitly handle cases where time andlocation of information are noisy or non-existent.

In the rest of this section we start by exploring general priorwork in the area of social event detection (SED). We thenhighlight techniques presented for the SED task in MediaEval2013 [1], the challenge at which this work was originallydesigned to compete, in Section V-B.

A. Prior work

In their system, Zaharieva et al. [25] attempt to achievedetection of specific social events as defined by a textual,temporal and geographic query. Their approach demonstratesa common pipeline approach wherein all multimedia itemsare clustered through a sequential divisions based on differentmetadata. Firstly, an initial clustering is attempted whichuses the temporal and spatial information of the multimediaitems. Further spatial clustering is then attempted by detectinggeographic words used in the image tags and description. Thedetected clusters are then compared to the specifications of thecluster query.

Petkos et al. [11] proposed a multimodal clustering ap-proach for the detection of social events. In their baselineapproach the authors suggest an aggregated affinity matrixspectral clustering approach similar to our spectral clusteringapproach. In their second approach the authors use pairwisesimilarities over all modalities to predict a “same cluster”classifier which takes a vector of distances from an imageto all other images as the classifier feature vector. Howeverboth their approaches are inherently incapable of scaling tolarger multimedia datasets. They either rely upon exhaustivelyholding all multimedia items to be clustered in memory, orthey expect a distance vector to be calculated for each itemto be clustered which incorporates a given items distance toall other items being clustered. Both demonstrate reasonableresults, but were not shown to work on sets of data larger than100,000 items.

Reuter and Cimiano [2] propose a more direct and scalablesolution to social event detection. They highlight a set of fea-tures which might commonly exist in social multimedia itemsincluding: time of capture, time of upload, geo location, title,description and tags. They then implement a procedure wherea database of seen items along with their event assignment areheld. When a novel document is to be assigned to an event, aninitial coarse query using searchable features such as textualcomponents and time is used to recieve a set of candidateitems. Custom distance metrics for each feature and a distancefusion strategy is then used to calculate a distance betweenthe new item and possible candidate items. Pre-trained SVMclassifiers are then used to decide whether the item is morelikely to belong to a new event, or whether it actually belongsto the event of one of the candidate items. This approachdoesn’t have the scale limitations present in Petkos et al. [11],however the use of a classifier to establish cluster membershiprelies explicitly on a representative training sample which

O P E N Q U E S T I O N S A N D F U T U R E D I R E C T I O N S

I S E V E N T D E T E C T I O N S O LV E D ?

• Scores in the 98% range certainly indicate so!

• But that’s a bit of a fallacy - the data was biased

• It only contained image of social events

• We need to do more exploration of real data streams which also contain noise in the form of images that don’t belong to a social event

W H Y D O N ’ T I M A G E F E AT U R E S W O R K W E L L I N T H E S E D TA S K ?

• The features we tried using were too strict.

• Designed for near duplicate detection

• but, social events are highly visually diverse

• highly visually similar images already being clustered through the other features

• Need to look at better engineered or more specific features

• Biometrics, Facial matching, Text, colour…

W H E R E N E X T W I T H T H E M U LT I M E D I A T W I T T E R A N A LY S I S ?

• Involve multimodal features

• Tweets are rather rich in data that can be combined with the image analysis

• Applications:

• Meme-tracking

• Detection of world events

• Exploiting multimodal feature co-occurrence

A C K N O W L E D G E M E N T S

David Dupplaw, Paul Lewis, Kirk Martinez, Jamie Davies, Neha Jain, John Preston, Laurence Wilmore, Ash Smith, Heather Packer and other members of the WAIS research group at the University of Southampton

A N Y Q U E S T I O N S ?

sewm'14 keynote: mining events from multimedia streams

Data & Analytics

e n g e s

e y n o t e

r i e s

n d t r e n d s i n

forgetr e s e

r e s h

n n o tat o r s

r e d o n t w