events in multimedia - theory, model, application

Events in Multimedia Ansgar Scherp – [email protected] Slide 1

Events in Multimedia -

Theory, Model, Applicat

ionWorkshop on Event-based Media Integration and

Processing, ACM Multimedia, 2013

Juniorprof. Dr. habil. Ansgar Scherp

[email protected]


Motivation

• Events are a natural abstraction

of human experience

• Events are everywhere!

• Lifelogs

• Experience sharing

• Emergency response

• Cultural heritage

• News

• News

• Sports

• Surveillance

• …


Theory


Brothers at enmity with each other …

Event

Object

Object

Event

• Earlier: object-based and entity-based systems

• Now: applications that consider events at least

as important as objects

vs.


What is an event?

• Perduring entities that unfold

over time

• Occurrences in which humans

participate

• Subject to discussions and

interpretations by humans

• Enduring entities that unfold

over spaceObject

Event



• Some philosophers consider objects as 4D

• Extend across time just as they do in space Casati R, Varzi A (2006) Events. Stanford encyclopedia of philosophy.

http://plato.stanford.edu/entries/events

Object

Event

http://plato.stanford.edu/entries/events


Event Object


• Events and objects as first class entities

• Events and objects require each other!

• For example, in DOLCE

‘is participant in’

‘has participant’

• … not necessarily!


Model


Events need to be modeled …• … and are useful in a variety of domains

• Lifelogs

• Multimedia-based experience sharing

• Emergency response

• Cultural heritage

• News

• Sports

• Surveillance

• …


Emergency

Control Center

Forward

Liaison

Officer

Documentary

support

Calls to report about

a power outage

Creates incident

with audio recording

Request to

report about a

flooded cellar

Reports

by taking

photos

etc.

Emergency Response

Coordination

Emergency

Hotline

Fire Department

Police Department

Coordinate and

keep up to

dateReport

and update

about the incident

Coordinate

and keep up

to date

Report and update

about the incident

Citizen

• Several emergency response entities are involved

• Using different event-based systems

• Common understanding of multimedia information

needed to efficiently communicate between ERs

Snapped pole image from:

http://www.dailymail.co.uk/

Emergency Response Scenario


Requirements to a Common Event

Model*• Participative aspect

• Temporal aspect

• Spatial aspect

• Structural aspect

• Mereology (composition)

• Causality

• Correlation

• Interpretation

• Experiential aspect (documentation)

* Analysis of 21 models and systems [SM13, SSF+12, SSF+09]


Survey on Event Models & Systems

…

Part

icip

a-

tion


Survey on Event Models & Systems…


Ontology Patterns of Event-Model-F• Event-Model-F* defines six ontology patterns

• Ontology design pattern similar to SE

• Build on top of DOLCE+DnS Ultralight ontology

• Cf. theory on events and objects

• Provides Description and Situation pattern

• Specified in Web Ontology Language (OWL)

• Formalized in Description Logics

* Homage to event model E by Westermann and Jain


6 Patterns of the Event-Model-F

• (1) Participation pattern

• (2) Mereology pattern (composition)

• (3) Causality pattern

• (4) Correlation pattern

• (5) Documentation pattern

• (6) Interpretation pattern

• All based on Descriptions and Situations (DnS)

• Contextualization of events and objects w.r.t.

specific situations


Example: Descriptions ‘n’ Situations• DomesticPowerOutage-

Description defines roles

• AffectedObjectRole

• AffectedPersonRole

• AffectedPersonRole

• …

• DomesticPowerOutage-

Situation defines objects

house-1 : Building

paul-1 : NaturalPerson

sandy-1 : NaturalPerson

…

• In a different situation, NaturalPerson

paul-1 may play a FireFighterRole

• People may have differing opinions

about the cause of the power outage

• …

Image source: Wikipedia

Classify


(1) Participation Pattern• Participation of living and non-living objects in events

• Reuse of domain knowledge

Roles the

entities play

Real world

entities

EventParticipationDescription

defines exactly 1 DescribedEvent

defines min 1 Participant

defines some LocationParameter

defines some TimeParameter

defines only (DescribedEvent or Participant or

LocationParameter or TimeParameter)

isSatisfiedBy exactly 1 EventParticipationSituation

Example: Firemen and home owner are involved in an

incident of a house fire.


(3) Causality Pattern

• Event (cause) implies other event (effect)

• Causal relationship holds under some justification

• Causes and effects are events, and only events

Description

EventCausalityDescription

EventCausalitySituation

Situation

Cause Effect Justification

EventRole

Concept

Event Description

classifiesisRoleOf

defines

isEventIncludedIn

satisfies

isObjectIncludedIn

Role

EventCausalityDescription

defines exactly 1 Cause

defines exactly 1 Effect

defines exactly 1 Justification

defines only (Cause or Effect or Justification)

isSatisfiedBy exactly 1 EventCausalitySituation

Example: The event of a snapped power pole causes a

power outage.


Use Case: Emergency Response

Domain ontology by

City of Sheffield, UK

Events and

media metadata

Event details

[MTAP2012]

Web 2.0 content

and metadata


Application


• Which bakery close by is open on Sunday?

• What could I do tonight?

• Which sights are in the area?

Are they still open?

• Existing applications

– Mostly focus on location,

i.e., points-of-interests

– No support for integrated

search for events Sun: 7am-11am

Problem: Events in Social Media


Different Sources of Social Events …• Opening hours of a shop like a bakery

• Concert of favorite rock star in my town

• Café serving English tea in my neighborhood

• Happy hour at a bar

• Special sales at the shopping mall

• …

…


Event Model for Social Media!

• Derive a specific model of social media events

• Use notions of events and objects defined in

Event-Model-F and specialize it for social media

• Focus on what is needed in the domain

– (1) Participation pattern

– (5) Documentation pattern

• Future work: (2) Mereology Pattern


• Exploration of POIs and

events in real-time

• Multithreaded integration

of social data sources:

KlickTel, DBpedia,

GeoNames, Eventful,

etc.

Pad

Phone

mobEx: Mobile Social Media Explorer


Mediator-based Architecture


Update Event 003Add Event 003

Incremental Data Integration on Server

mobEx

Server

Client

Café ViennaPhone.: ???

Opening hours:

Mo – Fr. 10 – 2 Uhr

Provider 1 Provider 2

Café ViennaPhone.: 01234

Opening hours: ???

Café

Vienna

Mo – Fr.

10 – 2

o‘clock

Café Vienna

Tel.: 01234

Mo – Fr. 10 – 2

o‘clock

Entity

Resolution

Time


Provider …

Entity Resolution in Detail

Provider 1

Provider 2

….

Weighted comparison of attributes:

Location information

Time information

Similarity score computed from

single weights

Above threshold integration

Café ViennaTel.: 01234

Opening hours:

Mo – Fr. 10 – 2 Uhr

S1, 15

68161 Mannheim

Calculated GPS Data

www.cafevienna.de/

Description: XYZ

High Weight

Low Weight

Medium Weight

High Weight

Low Weight

High Weight

Heuristic filter (blocking):

GPS coordinates > 500m

distance

Except: same address

Different entity types are not

compared (Persons, Places,

Events)

High Weight


Entity Resolution vs. Data Delivery

• Percentage of resources the client receives

• Percentage of resolved resolution at that time


Faceted Exploration of Events & Objects


Exploring Time as Natural as Space


• 18 subjects, average age of 25.5 (SD=3.03)

and six female

• Tracked detailed activities over three weeks

Evaluation in the Field.

• More than 4000 single events in 234 sessions

• Session is defined as

• Active usage of application until it is closed

• Or: after 30 seconds of inactivity


Evaluation in the Field (2)• Session lasted on average 2:42 minutes.

• 57% of the participants used the application

daily, 72% every second day or more.

• Users spent more time on the map screen than

on the screen showing the facets (ca. x1.5)

• Time slider not used

more often in the course

of the three weeks


Conclusion

• Events are an important concept in multimedia

• Tremendous research conducted in the past

• But, we have a lack of

– Common theory of events in multimedia

– Integrated tool chain to deal with events

– From detection over representation to use

Acknowledgements:

• R. Jain, C. Saathoff, T. Franz, S. Staab, D. Schmeiß

• B. Opitz, T. Sztyler, B. Pfister, M. Jess, C. Bikar, F. Knip


But there is a long interest …

• … in this topic in the multimedia community

• 1st ACM Int. Workshop on Events in Multimedia,

ACM Multimedia 2009, Beijing, China

• 2nd ACM Int. Workshop on Events in Multimedia,

ACM Multimedia 2010, Firenze, Italy

• 3rd ACM Int. Workshop on Events in Multimedia,

ACM Multimedia 2011, Scottsdale, Arizona, USA

• "Multimedia Activity and Event Understanding" Area,

ACM Multimedia 2012, Nara, Japan

• Workshop on Event-based Media Integration and

Processing, ACM Multimedia 2013, Barcelona


Try it out …in Barcelona!

events in multimedia - theory, model, application

Science