125

SITUATION RECOGNITION FROM MULTIMODAL DATA

Vivek K Singh1 Siripen Pongpaichet2 and Ramesh Jain2

1Rutgers University 2University of California Irvine

125

Todayrsquos slides

2

httpwwwspringercomusbook9783319305356Or email us for a softcopy

httpbitly29JL30M

125

Todayrsquos slides

2

httpwwwspringercomusbook9783319305356Or email us for a softcopy

httpbitly29JL30M

125

Course Outline1) Concept recognition from Multimedia data (20 mins Ramesh Jain)bull Trends bull Why situation recognition is different from object event scene recognition etc2) Situation recognition across multiple research domains (20 mins Vivek Singh)bull Situation Algebra Situation Calculus Robotics hellip3) Situation recognition (45 mins Vivek Singh) bull Situation modeling bull Situation operators 4) Designing situation based applications (30 mins Siripen Pongpaichet)bull Motivation and essential requirementsbull Application scenarios Thailand flood hurricane Sandy city sensing asthma relief5) Future trends and open problems (20 mins Ramesh Jain)bull Future trends bull Open problems for Multimedia research 6) Question and Answers (15 mins)

3

125

CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)

4

125

Introduction

bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event

scene recognition etc)

5

1256

Data Information Knowledge Wisdom

Data is Essential But we are really interested in products

Information Knowledge and Wisdom

1257

What is Important in lsquoBig Datarsquo

Multimedia

Realtime Uncertainty

1258

The Grand Challenge

Sense making from multimodal massive geo-social data-

streams

125

Fundamental Problem

Connecting People to Resources effectively efficiently and promptly

in given situations

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)

4

125

Introduction

bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event

scene recognition etc)

5

1256

Data Information Knowledge Wisdom

Data is Essential But we are really interested in products

Information Knowledge and Wisdom

1257

What is Important in lsquoBig Datarsquo

Multimedia

Realtime Uncertainty

1258

The Grand Challenge

Sense making from multimodal massive geo-social data-

streams

125

Fundamental Problem

Connecting People to Resources effectively efficiently and promptly

in given situations

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Introduction

bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event

scene recognition etc)

5

1256

Data Information Knowledge Wisdom

Data is Essential But we are really interested in products

Information Knowledge and Wisdom

1257

What is Important in lsquoBig Datarsquo

Multimedia

Realtime Uncertainty

1258

The Grand Challenge

Sense making from multimodal massive geo-social data-

streams

125

Fundamental Problem

Connecting People to Resources effectively efficiently and promptly

in given situations

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

1256

Data Information Knowledge Wisdom

Data is Essential But we are really interested in products

Information Knowledge and Wisdom

1257

What is Important in lsquoBig Datarsquo

Multimedia

Realtime Uncertainty

1258

The Grand Challenge

Sense making from multimodal massive geo-social data-

streams

125

Fundamental Problem

Connecting People to Resources effectively efficiently and promptly

in given situations

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

1257

What is Important in lsquoBig Datarsquo

Multimedia

Realtime Uncertainty

1258

The Grand Challenge

Sense making from multimodal massive geo-social data-

streams

125

Fundamental Problem

Connecting People to Resources effectively efficiently and promptly

in given situations

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

1258

The Grand Challenge

Sense making from multimodal massive geo-social data-

streams

125

Fundamental Problem

Connecting People to Resources effectively efficiently and promptly

in given situations

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Fundamental Problem

Connecting People to Resources effectively efficiently and promptly

in given situations

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12510

What is Cyber Space

Who invented it

Theory of Control and Communication in

Animals

Machines

Societies

Published first in 1942

Back to Future

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12511

Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)

bullCurrent State (using multimedia data)

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12512

In Smart Systems Feedback is the Key

InputComputed

using System Model

FeedbackOutput compared with

desired goal

Actual System

OutputObserved

Continuously

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Social Networks

Connecting People

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12505032023 14

Connecting People And

Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12515

Traditional Social SystemsbullModels of Systems were difficult to form

bullCurrent State of the system could not be determined

bullReal time lsquoactionsrsquo could not be implemented

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12516

Emerging Social Systems

bullSocial models can be determined using warehouses of Big Data

bullSocial observations are now possible with little latency

bullActions could be targeted to precise sources

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12517 EventShop Global Situation Detection

Predictive Situation

Recognition

Evolving Global Situation

Predictive Personal Situation

Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

PersonaDatabase

Resources

Needs

Data Ingestio

n

Wearable Sensors

Calendar

Locationhellip

Dat

a So

urce

s

hellip

Data Ingestion

and aggregatio

n

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12518

Concept Recognition Last Century

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACETIME

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Visual Concept Recognition First research papers

bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object SceneTrajectory

Event

Situation

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12520

Concept Recognition This Century

Environments

Real world Objects

Situations

Activities

SPACETIME

REAL-

WORLD

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just DataMedium and sources do not matter

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12521

Concept recognition from multimedia data

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Situations

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 114K

34 KLocation aware

Location unaware

Static Dynamic

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Situation Recognition Next Frontierbull Data Abundance

bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams

bull Need new frameworks

22

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS

(20 mins Vivek Singh)

23

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12524

Related Work Data to SituationsArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processingActive DB

X X o X

GIS X o X X o

Mashup toolkits(Y pipes ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

XThis work X X X X X X X

XX

o = partial support

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Defining Situations Situation Calculus

bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968

bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors

for undertaking control decision ndash Singh amp Jain 2009

25

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)

stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))

EventsFluents

isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control

isInRoom(P1 s) 0

isWorking(P1 s) 01

1 Situation

Situation = Not events nor sequence of events but their assimilated descriptor

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Problems with this approachbull Scalability

bull Listing all the rules bull Frame problem - Specifying the non-effects

bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data

27

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Situationsbull Multiple definitions

bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing

ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo

ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)

ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo

ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)

ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12529

Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions

bull Abstractionbull Computationally

Grounded

Work Goal Based Space-Time Future Actions Abstraction Computationally

GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o

This work (aim) X X X X Xo = Partial support

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12530

Situation Definition

bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road

congestion wildfire flash-mob

Goal Based Space-Time Future Actions Abstraction Computationally

Grounded

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Overall Framework Motivating example

31

STT data

TweetlsquoUrrghhellip sinusrsquo

Loc NYCDate 3rd Jun 2011

Theme Allergy

Situation Detection User-Feedback

lsquoPlease visit nearest CDC center at 4th St

immediatelyrsquo

Date 3rd Jun 2011

Aggregation

1) Classification2) Control action

Operations

Alert level = High

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12532

Applicationsbull Healthcare

bull Alert me if there is a flu epidemic in my areabull Telepresence

bull Which camera feed to send outbull Business analysis

bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather

bull Alert me when the fall colors blossom in New England bull Daily living

bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

SITUATION RECOGNITION FRAMEWORK

(45 mins Vivek Singh)

33

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

A) Situation Modeling

B) Situation Recognition

C) Visualization Personalization and Alerts

hellip hellip

STT Stream

Emage

Situation

hellipC1

v2 v3

v5 v6

prod

Δ

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 34

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12535

A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic

bull Building blocks bull Operators bull Operands

bull Wizard bull A prescriptive approach for modeling situations using

the operators and operands

Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12536

Growth rate (Flu reports) Feature

Thresholds (0 50)

Data source

Meta-data

-Emage (Reports)

Representation level

Twitter-Flu

Building Blocks Operandsbull Knowledge or data driven building blocks

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Building Blocks Operators

Δ Transform hellipSpatio-temporal

window

37

Aggregate +

Classification Classification method

Characterization Growth Rate = 125

Property required

Pattern Matching

72+

Pattern

prod Select +Mask

Φ Learn Learning method

Features

Situation

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12538

Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features

v=f(v1 hellip vk)bull If (type = imprecise)

bull identify learning data source method

4) ForEach (feature vi) If (atomic)

bull Identify Data source bull Type URL ST bounds

bull Identify highest Rep level reqdbull Identify operations

Else Get_components(vi)

vf1

v4

v2 v3

D1

Emage

Δ

D2

prod

Emage

Δ

D3

Δ

Emage D2

prod

Emage

Δ

f2

v5 v6

ltUSA 5 mins001x 001gt

ϵ Low Mid High

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Epidemic Outbreaks

Unusual Activity Growth Rate

Current activity level

Historical activity level

Emage (reports ILI)

Δ

Twitter-Flu

TwittercomltUSA 5 mins

001x 001gt

Emage (Historical avg)

Δ

Twitter-Avg

DB ltUSA 5 mins

001x 001gt

Δ

Twitter-Flu

Emage (reports ILI)

TwittercomltUSA 5 mins

001x 001gt

ϵ Low mid highltUSA 5 mins 001x

001gt

Growing Unusual activity

1)Model

Emage (reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

TwittercomltUSA 5 mins

001x 001gt

Censusgov ltUSA 5 mins

001x 001gt

2) Revise

Subtract

Subtract

Multiply

Classification Thresh (3070)

Normalize[0100]

3) Instantiate39

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12540

Level 1 Unified representation

(STT Data)

Level 3 Symbolic rep (Situations)

Properties

Properties

Properties

Level 0 Raw data streams eg tweets cameras traffic weather hellip

Level 2 Aggregation

(Emage)

hellip hellip hellip

STT Stream

Emage

Situation

B) Situation evaluation Workflow

Operations

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Data Representationbull E-mage

bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators

(eg segmentation background subtraction convolution)

41

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Data Modelbull Spatio-temporal element

bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage

bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set

bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set

bull TPS = (t1 p1) (tn pn)

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12543

Situation Recognition Algebra

Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10

S No Operator Input Output

1 Filter prod Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

Temporal E-mage Stream Temporal Pixel Stream

Temporal Pixel Stream Temporal Pixel Stream

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Media processing engine

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Implementation and resultsbull Twitter feeds

bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)

and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds

bull API bull Tags RGB values from gt800K images

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Testing Data Representation + Algebra

bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics

bull Data bull Twitter feeds archive

bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs

bull Flickr feedsbull API Tags RGB values from gt800K images

bull Implementation bull Matlab + Java + Python

46

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12547

Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo

bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)

bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))

bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))

bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina)))))

bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])

(TEStheme=Katrina))))

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12548 ATampT retail locations

ATampT total catchment area

iPhone theme based e-mageJun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39 -122] just north of

Bay Area CA

SpatialMax

ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt

+ Add

to Jun 15 2009

Convolution

Store

catchment area

Convolution

Store catchment

area

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Flickr Social Emagesbull Jan ndash Dec 2009

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

12550

Seasonal characteristics analysisbull Fall colors in New England

bull Show me the difference between red and green colors for New England region as it varies throughout the year

bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)

(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))

Jan

0

Dec

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

Year average Peak of green

At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park

125

OTHER APPROACHES

52

125

FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics

Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies

Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics

125

Proposed Approach54

03052023

Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies

Model visual analytics concepts bull pixelsbull frame

Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users

Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)

125

FraPPE visually55

03052023

Time

Reality

Capture

Frame

Digital Reflex

125

FraPPE visually56

03052023

Grid

Cell

Time

Frame

125

FraPPE visually57

03052023

Pixel Frame 1

Time

125

FraPPE visually58

03052023

Place A

Event A

Time

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics

Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies

Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics

125

Proposed Approach54

03052023

Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies

Model visual analytics concepts bull pixelsbull frame

Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users

Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)

125

FraPPE visually55

03052023

Time

Reality

Capture

Frame

Digital Reflex

125

FraPPE visually56

03052023

Grid

Cell

Time

Frame

125

FraPPE visually57

03052023

Pixel Frame 1

Time

125

FraPPE visually58

03052023

Place A

Event A

Time

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Proposed Approach54

03052023

Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies

Model visual analytics concepts bull pixelsbull frame

Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users

Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)

125

FraPPE visually55

03052023

Time

Reality

Capture

Frame

Digital Reflex

125

FraPPE visually56

03052023

Grid

Cell

Time

Frame

125

FraPPE visually57

03052023

Pixel Frame 1

Time

125

FraPPE visually58

03052023

Place A

Event A

Time

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

FraPPE visually55

03052023

Time

Reality

Capture

Frame

Digital Reflex

125

FraPPE visually56

03052023

Grid

Cell

Time

Frame

125

FraPPE visually57

03052023

Pixel Frame 1

Time

125

FraPPE visually58

03052023

Place A

Event A

Time

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

FraPPE visually56

03052023

Grid

Cell

Time

Frame

125

FraPPE visually57

03052023

Pixel Frame 1

Time

125

FraPPE visually58

03052023

Place A

Event A

Time

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

FraPPE visually57

03052023

Pixel Frame 1

Time

125

FraPPE visually58

03052023

Place A

Event A

Time

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

FraPPE visually58

03052023

Place A

Event A

Time

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

FraPPE visually59

03052023

Event A

Time

Frame 1

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

FraPPE visually60

03052023

Event B

Place B

Time

Frame 2

Frame 1

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

12561

City Sensing listens to the pulse of Milano Design Week on April 9th 2014

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

12562

Tweeting Cameras

Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli

Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Physical amp Social Sensors Fusion For Situation Awareness

PhysicalSensors

SocialSensors

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Real-world Events

Hispanic ParadeCBGB Musical Festival

Columbus Day Parade

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

twtw

tw tw

tw tw

Historic TweetsRecent Tweets

Event time and location

Retrieve recent tweets Retrieve historic tweets

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

12565

Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The

fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo

bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value

bull pointer points to actual raw data stream

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Physical Sensors (Concept ldquoCrowdrdquo)

>

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Social Sensors (MillionMarchNYC BlackLivesMatter)

>

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Fused Information

>

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

CMage1

09

08

07

06

05

04

03

02

01

0

Gaussian Process based Prediction

Sensor Image Patch

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Fusing Sensor Cmage with Social Cmage

Sensor Cmage(concept ldquopeople marchingrdquo)

Social Cmage(concept ldquoMillionsMarchNYCrdquo)

Fused Cmage

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)

71

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion

72

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

EventShop Requirement

73

Granulari-ties

Heterogeneous

Model Prediction

UsersOpen-Source

Storage

Generic

Streams

Support fast data flow

Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities

Provide storage system to archive both data input and system outputCreate situation model and provide actionable information

Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

Contain predictive component

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

EventShop Architecture74

AlertOutputData Ingestor

Data Source Parser

Data Adapter

Emage Generator

(+resolution mapper)

Query Processing

EventShop Storage

Query Parser

Query Rewriter

Emage Stream Processing

Action Parser

Situation

Emage

Visualization (Dashboard)

Actuator Communication

Event Property amp Other Information

(eg spatio-temporal pattern)

ᴨ

ᴨmicro

Data Access Manager

Live StreamArchived Stream

Situation Stream

Physical Data Source (eg sensor

streams geo-image streams)

Logical Data Source (eg preprocessing data streams social

media streams)

Raw Event

REST API ServicesData Source Query Alerts STT-Emage

EventShop UI External AppsVisual Analytics

Inte

rface

AP

IP

roce

ssin

g La

yer

Sto

rage

Lay

er

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

EventShop UI

75

Save Query Reset Query

Create Query

Pollen

Tweets_Asthma

Available

Available

350

AQI Available 357

361

Grouping Stopped 35

Asthma_ Risk

Stopped 36

Asthma_ Interpolate

Stopped 37

Asthma_ Interpolate

Stopped 37

Asthma_ Stopped 37

Query Graph

Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char

Redraw

ds361

ds350

ds357

F1 Q1

F2

F3

Q2

Q3

A1 Q4 G1 Q5

httpeventshopicsuciedu8080eventshoplinux

125

Demo

76

httpswwwyoutubecomwatchv=E5unHXZmSr8

125

Demo

77

httpswwwyoutubecomwatchv=IwJYEZd8Bbg

>

125

Building applications using EventShopSNo Application Data Used Application

deployed Scale Data modalities Operators used

1 Wildfire detection in California Real Yes Macro Satellite data

Google insights F A Ch

2 Hurricane monitoring Simulated No Macro na F A Ch P

3 Flu epidemic surveillance Real No Macro Twitter Census F A C

4 Allergy recommendation Real Yes Macro Twitter Air Quality Pollen Count F A C

5 Asthma management Real Yes Macro Personalized alerts

In situ sensors Satellite data

Asthma Tracking F I Pr

6 Thailand flood mitigation Real Yes Macro Personalized alerts KML F A C

7 Photos as Micro-Reports Real Yes Macro Flickr F Cl

8 Trash management Real amp Simulated In progress Macro Trash sensors

micro-reports F A Pr

LegendF = Filter A = AggregateC = Classification Ch = CharacterizationI = Interpolation Cl = ClusterP = Pattern MatchingPr = Prediction

78

125

Asthma ManagementApplication

79

125

Asthma Management Application

80

(1) Macro

Situation

Macro Data Streams

(3) Situation-

Action Rules

Sensor streams

Social media

Geo-temporal data

Personal Data Streams

(2) Personal Situation

Behavioral streams

Profile + Preferences

125

Asthma Risk Estimation

81

Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration

visualize data on Feb 12th 2008

Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015

Spectral Spatial Gaussian Process (SSGP)

125

Experiment Results

82

Data Model PMSE MAPE

SingleData Source

CMAQ - 10619 272873CMAQ LR 09586 271077

Stations Kriging 09077 229672CMAQ SSGP 03468 142727

MultipleData Sources

ALL SGP 03006 135109ALL SSGP 02858 131087

CMAQ Kriging SSGP

125

Asthma Risk Estimator Model and Result

83

Asthma HospitalizationGround Truth

FILTERLOC=CA

FILTERLOC=CA

AGGFUNC=AVG

GROUPTHRESHOLD

Asthma Risk Area without Interpolation

GROUPTHRESHOLD

Asthma Risk Area with interpolation

AGGFUNC=AVG

PM25 ConcentrationFrom Stations

InterpolatedPM25 using SSGP

Pollen

OzoneAQI

125

A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK

Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1

1University of California Irvine USA2Northwestern Polytechnical University China

Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Asthma ManagementApplication

79

125

Asthma Management Application

80

(1) Macro

Situation

Macro Data Streams

(3) Situation-

Action Rules

Sensor streams

Social media

Geo-temporal data

Personal Data Streams

(2) Personal Situation

Behavioral streams

Profile + Preferences

125

Asthma Risk Estimation

81

Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration

visualize data on Feb 12th 2008

Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015

Spectral Spatial Gaussian Process (SSGP)

125

Experiment Results

82

Data Model PMSE MAPE

SingleData Source

CMAQ - 10619 272873CMAQ LR 09586 271077

Stations Kriging 09077 229672CMAQ SSGP 03468 142727

MultipleData Sources

ALL SGP 03006 135109ALL SSGP 02858 131087

CMAQ Kriging SSGP

125

Asthma Risk Estimator Model and Result

83

Asthma HospitalizationGround Truth

FILTERLOC=CA

FILTERLOC=CA

AGGFUNC=AVG

GROUPTHRESHOLD

Asthma Risk Area without Interpolation

GROUPTHRESHOLD

Asthma Risk Area with interpolation

AGGFUNC=AVG

PM25 ConcentrationFrom Stations

InterpolatedPM25 using SSGP

Pollen

OzoneAQI

125

A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK

Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1

1University of California Irvine USA2Northwestern Polytechnical University China

Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Asthma Management Application

80

(1) Macro

Situation

Macro Data Streams

(3) Situation-

Action Rules

Sensor streams

Social media

Geo-temporal data

Personal Data Streams

(2) Personal Situation

Behavioral streams

Profile + Preferences

125

Asthma Risk Estimation

81

Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration

visualize data on Feb 12th 2008

Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015

Spectral Spatial Gaussian Process (SSGP)

125

Experiment Results

82

Data Model PMSE MAPE

SingleData Source

CMAQ - 10619 272873CMAQ LR 09586 271077

Stations Kriging 09077 229672CMAQ SSGP 03468 142727

MultipleData Sources

ALL SGP 03006 135109ALL SSGP 02858 131087

CMAQ Kriging SSGP

125

Asthma Risk Estimator Model and Result

83

Asthma HospitalizationGround Truth

FILTERLOC=CA

FILTERLOC=CA

AGGFUNC=AVG

GROUPTHRESHOLD

Asthma Risk Area without Interpolation

GROUPTHRESHOLD

Asthma Risk Area with interpolation

AGGFUNC=AVG

PM25 ConcentrationFrom Stations

InterpolatedPM25 using SSGP

Pollen

OzoneAQI

125

A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK

Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1

1University of California Irvine USA2Northwestern Polytechnical University China

Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Asthma Risk Estimation

81

Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration

visualize data on Feb 12th 2008

Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015

Spectral Spatial Gaussian Process (SSGP)

125

Experiment Results

82

Data Model PMSE MAPE

SingleData Source

CMAQ - 10619 272873CMAQ LR 09586 271077

Stations Kriging 09077 229672CMAQ SSGP 03468 142727

MultipleData Sources

ALL SGP 03006 135109ALL SSGP 02858 131087

CMAQ Kriging SSGP

125

Asthma Risk Estimator Model and Result

83

Asthma HospitalizationGround Truth

FILTERLOC=CA

FILTERLOC=CA

AGGFUNC=AVG

GROUPTHRESHOLD

Asthma Risk Area without Interpolation

GROUPTHRESHOLD

Asthma Risk Area with interpolation

AGGFUNC=AVG

PM25 ConcentrationFrom Stations

InterpolatedPM25 using SSGP

Pollen

OzoneAQI

125

A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK

Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1

1University of California Irvine USA2Northwestern Polytechnical University China

Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Experiment Results

82

Data Model PMSE MAPE

SingleData Source

CMAQ - 10619 272873CMAQ LR 09586 271077

Stations Kriging 09077 229672CMAQ SSGP 03468 142727

MultipleData Sources

ALL SGP 03006 135109ALL SSGP 02858 131087

CMAQ Kriging SSGP

125

Asthma Risk Estimator Model and Result

83

Asthma HospitalizationGround Truth

FILTERLOC=CA

FILTERLOC=CA

AGGFUNC=AVG

GROUPTHRESHOLD

Asthma Risk Area without Interpolation

GROUPTHRESHOLD

Asthma Risk Area with interpolation

AGGFUNC=AVG

PM25 ConcentrationFrom Stations

InterpolatedPM25 using SSGP

Pollen

OzoneAQI

125

A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK

Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1

1University of California Irvine USA2Northwestern Polytechnical University China

Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Asthma Risk Estimator Model and Result

83

Asthma HospitalizationGround Truth

FILTERLOC=CA

FILTERLOC=CA

AGGFUNC=AVG

GROUPTHRESHOLD

Asthma Risk Area without Interpolation

GROUPTHRESHOLD

Asthma Risk Area with interpolation

AGGFUNC=AVG

PM25 ConcentrationFrom Stations

InterpolatedPM25 using SSGP

Pollen

OzoneAQI

125

A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK

Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1

1University of California Irvine USA2Northwestern Polytechnical University China

Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK

Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1

1University of California Irvine USA2Northwestern Polytechnical University China

Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Detecting Situations from Micro-Reports

85

Photos

Reports Events

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

PHOTOS as Kodak Moments

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Disruption PHOTOS as Information

Smartphone camera captures

EVENTS

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability

Reports of Events from Journalists

Seek Truth and Report it as Fully as Possible

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Reports of Events from Citizens

>

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

FASTSubjective

EASY Noisy

LATESTAmbiguous

were SO yesterdayMicro-Blogs

Multimedia Micro-Reports

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Compelling Universal

Objective Spontaneous

Multimedia Micro-Reports (MMRs) are now and future

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Capturing and Reporting events using multimedia such as photos videos sensors and texts

Converting multimedia data to multimedia micro-reports using MediaJSON

Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics

Emerging opportunities for numerous apps smart city public health emergency rescue

What are the challenges

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

What ObjectsWho PeopleWhen EventsWhere Location

Why IntentEmotions

How Photo and audio

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Capturing and Reporting Events with Krumbs SDK

httpskrumbsnet

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Real-time MMR Dashboard

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Converting Multimedia Data into MMR

micro_reports[ where

geo_location latitude3290233332316081 longitude-1172441166718801

whenstart_timeJun 14 2009 112519

AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles

what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004

visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]

Photo

What

Where

When

Who

Why

Sound

MediaJSON

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

MediaJSONData

Wrapper

Data Wrapper

Data Wrapper

Data Wrapper

IoT(Event-driven operation)

Converting Multimedia Data into MMR

MediaJSONData

Wrapper

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Number of photos in London per day

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Evolving Photo Concepts in London

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo

Detecting Olympic Games

Olympic Games = basketball court game gymnastics people sport stadium swim tennis

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Detecting London Olympic GamesSummer Olympic Game

in July and August

Paralympic Games in September

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

bull Temporal range 1 year from July 2011 to June 2012

bull Location Thailand

Detecting Emergency Situations

City flood = outdoor water road car

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Photos from City Flood Cluster

November 5 2011

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Smart City Project in DC

105

US Presidential Inauguration in DC

Earth Day Concert

Cherry Blossom Festival

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Integrating MMR with other data sources for Situation Recognition (In progress)

h t t p s m a r t c i t i e s i n n o v a t i o n c o m

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125107

Trash Fill Level Situation in DC

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

730 800 830 900 930 1000 10300

20

40

60

80

100

0

3550

90

Real-Time Fill Level Situations at a given location of an event

Prediction based on Events HistoryEvents Data

Real-Time Trash Fill Level Situation

730 800 830 900 930 1000 10300

20

40

60

80

100

120

1020

40

70

90100

20

0

35

50

90

Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level

20 42

Now

Predicted Trash Fill Level in 30 minutes at a given location

78 99

30 minutes

730 800 830 900 930 1000 10300

20406080

100120

1030

40

7090

100

20

Projected Trash Fill Level at a given location based on Event History

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial

videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai

Siripen Pongpaichet (spongpaiuciedu)

109

125

FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)

110

125

Future Trends

bull Future trends bull Open problems for Multimedia research

111

125

This century is different from the last

Should we think differently

125In 20th century we tolerated

photos in our textual documents

In 21st century you create visual documents that tolerate text

125Major Disruption in Photos From Memories to Information Sources

Photos are the most compelling source of information

125115

125116

We are immersed in Big Data

Multimedia

Realtime Uncertainty

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125

Future Trends

bull Future trends bull Open problems for Multimedia research

111

125

This century is different from the last

Should we think differently

125In 20th century we tolerated

photos in our textual documents

In 21st century you create visual documents that tolerate text

125Major Disruption in Photos From Memories to Information Sources

Photos are the most compelling source of information

125115

125116

We are immersed in Big Data

Multimedia

Realtime Uncertainty

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125

This century is different from the last

Should we think differently

125In 20th century we tolerated

photos in our textual documents

In 21st century you create visual documents that tolerate text

125Major Disruption in Photos From Memories to Information Sources

Photos are the most compelling source of information

125115

125116

We are immersed in Big Data

Multimedia

Realtime Uncertainty

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125In 20th century we tolerated

photos in our textual documents

In 21st century you create visual documents that tolerate text

125Major Disruption in Photos From Memories to Information Sources

Photos are the most compelling source of information

125115

125116

We are immersed in Big Data

Multimedia

Realtime Uncertainty

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125Major Disruption in Photos From Memories to Information Sources

Photos are the most compelling source of information

125115

125116

We are immersed in Big Data

Multimedia

Realtime Uncertainty

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125115

125116

We are immersed in Big Data

Multimedia

Realtime Uncertainty

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125116

We are immersed in Big Data

Multimedia

Realtime Uncertainty

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125117

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125

Data as a Platformbull Multi-modal

bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for

118

Connecting People to Resources effectively efficiently and promptly

in given situations

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125

Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull

bull Siripen Pongpaichetbull spongpaiicsuciedu

bull Ramesh Jainbull jainicsuciedu

119

125

Useful linksbull Copies of publications

bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides

bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf

bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos

presentations and publications httpwwwicsuciedu~spongpai bull Related Projects

bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http

wwwstreamreasoningorglivefestivalcomunicazione2014

120