/ 46

PERSONALIZED SITUATION RECOGNITION

Vivek K. Singh

Information Systems Group,

University of California, Irvine

Advised by: Professor Ramesh Jain

/ 462

Trends

SOCIAL

GEO-SOCIAL

/ 463

Trends

PLANETARY SCALE

/ 464

Trends

SENSE MAKING

/ 465

The Big Challenge

Sense making from planetary scale geo-social data-streams

Situation recognition

/ 466

Concept recognition from multimedia data

Environments

Real world Objects

Situations

Activities

Heterogeneous M

edia

Single M

edia

SPACETIME

REAL-

WORLD

ScenesLocation aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Heterogeneous M

edia

Single M

edia360 K 11.4K

3.4 KLocation aware

Location unaware

Static Dynamic

/ 467

Contributions

1. Computationally define situations

2. Define a generic process for Situation recognition

a) Situation Modeling

b) Situation Evaluation: • E-mage + Situation Recognition Algebra

c) Personalized Alerts

3. EventShop: Web-based system for situation evaluation

/ 468

Situations: Other definitions•Endsley, 1988: “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future”

•Merriam-Webster dictionary: “relative position or combination of circumstances at a certain moment”

•McCarthy, 1969: “A situation is a finite sequence of actions.”

•Yau, 2006: “A situation is a set of contexts in the application over a period of time that affects future system behavior”

•Dietrich, 2003: “…extensive information about the environment to be collected from all sensors independent of their interface technology. Data is transformed into abstract symbols. A combination of symbols leads to representation of current situations…which can be detected”

/ 469

Situations: commonalities• Goal Based• Space-Time• Future Actions

• Abstraction• Computationally

Grounded

Work Goal Based Space-TimeFuture

ActionsAbstraction 

Computationally Grounded

McCarthy, 1968     X    Barwise, 1971   X   X  Endsley, 1988   X X X XSarter, 1991   o   X  Adam, 1993 X   X    Dominguez,1994 X   X X XSmith, 1995 X o X X  Steinberg, 1999 X   X X oJeannot, 2003 X        Moray, 2004   o   X  Dietrich, 2004       X XYau, 2006 X   X   XDostal, 2007   o   X  Singh, 2009 X   X   XMerriam-Webster (accessed 2012)   o      

This work (aim) X  X X  X X

o = Partial support

/ 4610

Situation: Definition

• Situation: An actionable abstraction of observed spatio-temporal characteristics.• e.g. flu epidemic, severe asthma threat, road

congestion, wildfire, flash-mob

Goal Based Space-TimeFuture

ActionsAbstraction 

Computationally Grounded

/ 46

Overall Framework: Motivating example

11

STT data

Tweet:‘Urrgh… sinus’

Loc: NYC,Date: 3rd Jun, 2011

Theme: Allergy

Situation Detection User-Feedback

‘Please visit nearest CDC center at 4th St

immediately’

Date: 3rd Jun, 2011

Aggregation,

1) Classification

2) Control action

Operations

Alert level = High

/ 46

Eco-system: Situation based applications

Singh, Jain: Situation based control. (Best Student Paper) IEEE Situation Management Workshop’09Singh, Kankanhalli, Jain: Motivating contributors. (Best Paper) ACM Workshop on Social Media ’09

Situation based

controller

Analysis & insights

Control decisions

Alerts

Spatio-Temporal

aggregation

Macro situation

App logicAnalyst

Personal situation

Situation detection operators

Event processing engine

Device Sensors

Archives

Human Sensor/ Wisdom source

Human Sensor/Actuator

12

/ 46

A) Situation Modeling

B) Situation Recognition

C) Visualization, Personalization, and Alerts

… …

STT Stream

Emage

Situation

…C1

v2 v3

v5 v6

@

∏

Δ@

i) Visualization

ii) Personalization

+

+Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 13

/ 4614

Design principles• Humans as sensors• Space + Time as fundamental axes • Real time situation evaluation (E-mage Streams)

(a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream)

d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)

/ 4615

A) Situation Modeling

• Help domain experts externalize their internal models of situations of interest e.g. epidemic.

• Building blocks: • Operators • Operands

• Wizard: • A prescriptive approach for modeling situations using

the operators and operands

Singh, Gao, Jain: Situation recognition: An evolving problem for heterogeneous dynamic big multimedia data, ACM Multimedia ‘12.

/ 4616

Growth rate (Flu reports) Feature

Thresholds (0, 50)

Data source

Meta-data

-Emage (#Reports)

Representation level

Twitter-Flu

Building Blocks: Operands• Knowledge or data driven building blocks

/ 46

Building Blocks: Operators

Δ Transform …Spatio-temporal

window

17

Aggregate +

ClassificationClassification

method

@ Characterization Growth Rate = 125%

Property required

Pattern Matching

72%+

Pattern

∏ Filter +Mask

Φ Learn Learning method

{Features}

{Situation}

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s)

Data OutputOperator Type

/ 4618

Situation ModelingGet_components (Situation v){1) Identify output state space2) Identify S-T bounds3) Define component features:

v=f(v1, …, vk)• If (type = imprecise)

• identify learning data source, method

4) ForEach (feature vi) {

If (atomic)• Identify Data source.

• Type, URL, ST bounds • Identify highest Rep. level reqd.• Identify operations

Else Get_components(vi)

} }

vf1

v4

v2 v3

@

D1

Emage

Δ

D2

∏

Emage

Δ

D3

Δ

@

Emage D2

∏

Emage

Δ

f2

v5 v6

<USA, 5 mins,0.01x 0.01>

ϵ { Low, Mid, High}

/ 46

Epidemic Outbreaks

Unusual Activity? Growth Rate

Current activity level

Historical activity level

Emage (#reports ILI)

Δ

Twitter-Flu

Twitter.com<USA, 5 mins,

0.01x 0.01>

Emage (Historical avg)

Δ

Twitter-Avg

DB, <USA, 5 mins,

0.01x 0.01>

Δ

Twitter-Flu

Emage (#reports ILI)

Twitter.com<USA, 5 mins,

0.01x 0.01>

ϵ {Low, mid, high},<USA, 5 mins, 0.01x

0.01>

Growing Unusual activity

1)Model

Emage (#reports ILI)

Δ

Twitter-Flu

Emage (population)

Δ

CSV-Population

π

Twitter.com<USA, 5 mins,

0.01x 0.01>

Census.gov, <USA, 5 mins,

0.01x 0.01>

2) Revise

Subtract

Subtract

Multiply

Classification: Thresh (30,70)

Normalize[0,100]

3) Instantiate

19

/ 4620

Level 1: Unified representation

(STT Data)

Level 3: Symbolic rep. (Situations)

Properties

Properties

Properties

Level 0: Raw data streams e.g. tweets, cameras, traffic, weather, …

Level 2: Aggregation

(Emage)

… … …

STT Stream

Emage

Situation

B) Situation evaluation: Workflow

Operations

/ 46

Data Representation• E-mage

• Visualization• Spatio temporal data representation• Data analysis using media processing operators

(e.g. segmentation, background subtraction, convolution)

21

/ 4622

Data Representation

• Spatio-temporal element• STTPoint = {s-t-coord, theme, value, pointer}

• E-mage• g = (theme, x, v(x) | x ϵ X = R2 , and v(x) ϵ V = N)

• Temporal E-mage Stream• TES=((t0, g0), ..., (tk, gk), …)

• Temporal Pixel Stream• TPS = ((t0, p0), ..., (tk, pk), …)

/ 46

Situation Recognition Algebra

23

Pattern Matching

Aggregate

@ Characterization

∏ Filter

Classification

72%

+

+

Growth Rate = 125%

Data Supporting parameter(s) OutputOperator Type

+

Classification method

Property required

Pattern

Mask

/ 4624

Situation Recognition Algebra

Singh, Gao, Jain: Social Pixels: Genesis and Evaluation, ACM Multimedia ‘10.

S. No Operator Input Output

1 Filter ∏ Temporal E-mage Stream Temporal E-mage Stream

2 Aggregation K*Temporal E-mage Stream Temporal E-mage Stream

3 Classification Temporal E-mage Stream Temporal E-mage Stream

4 Characterization : @ Spatial Temporal

  Temporal E-mage Stream Temporal Pixel Stream

  Temporal Pixel Stream Temporal Pixel Stream

5 Pattern Matching Spatial Temporal

  Temporal E-mage Stream Temporal Pixel Stream

  Temporal Pixel Stream Temporal Pixel Stream

/ 4625

Sample Queries

• Select E-mages of USA for theme ‘Obama’.• ∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama)

• Identify three clusters for each E-mage above.• kmeans(3) (∏spatial(region=[24,-125],[24,-65])(TEStheme=Obama))

• Show me the cluster with most interest in ‘Obama’.• ∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama)))

• Show me the speed for high interest cluster in ‘Katrina’ emages• @speed(@epicenter(∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65])

(TEStheme=Katrina)))))

• How similar is pattern above to ‘exponential increase’?• exp-increase(@speed(@epicenter (∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65])

(TEStheme=Katrina))))

/ 4626

C) Personalization and Alerts

1) Macro situation

Macro data-sources

Personal Context

Profile + Preferences

2) Personalized

situation

User data

IF user Ui <is-in> (PSj) THEN <connect-to> Rk

Personalized situation: An actionable integration of a user's personal context with surrounding spatiotemporal situation.

3) Personalized

alerts

Available resources

Resourcedata

/ 46

Personalized Situation Recognition: Operators

Pattern Matching

Aggregate

@ Characterization

∏ Filter

Classification

+

+

Growth Rate = 125%

Data Supporting parameter(s) OutputOperator Type

+

Classification method

Property required

Pattern

User location

…

… … …

… …

…

…Match= 42%

27

/ 4628

Situation Action Rules

• U = Users • PS = Personalized Situations• R = Resources

/ 4629

EVENTSHOP:Recognizing situations from web streams

/ 4630

EventShop: System Implementation

• Front end: • Javascript (JSLinb library)

• Front-Back end Interaction• Java servlets, Apache

• Back End• Java• C++ (OpenCV classes)

• Ingestion wrappers available for:• Twitter streams, Flickr stream, CSV data, KML data, Geo-images,

MySQL data archives, Funf (mobile phone sensors)

Gao, Singh, Jain: EventShop: From Heterogeneous Web Streamsto Personalized Situation Detection and Control, ACM WebScience ‘12.

/ 4631

S.No Query Language Operator Media processing Operator

Media processing Operator Details

1. Filter

-Spatial Arithmetic AND with the spatial mask

-Temporal Arithmetic AND with the temporal mark

-Thematic Arithmetic =

-Value Arithmetic AND, >, <, =

2. Aggregation

-Max, Min, +,-,%,* Arithmetic Max, Min, +,-,%,*

- NOT, OR, AND, Logical NOT, OR, AND

-Convolution Convolution Convolution

3. Classification

- Predefined segments count Segmentation K-means

- Predefined segment boundaries Segmentation thresholds

4. Characterization

i) Spatial

- Count, Min, Max, Sum, Average, Variation Statistical Count, Min, Max, Sum, Average

- Coverage Arithmetic Count

- Epicenter Arithmetic Weighted average

- Circularity Convolution Scale free convolution with known circular kernel

- Growth rate Arithmetic +, -, %

ii) Temporal

- Displacement, Distance, Velocity, Acceleration, Growth rate

Arithmetic +, -, %, *

- Future estimation Arithmetic Multiplication with Kernels based on users choice e.g. linear, progression exponential growth

- Periodicity Convolution Auto correlation i.e. Self convolution with time-lagged variant.

5. Pattern Matching

- Scaled Matching Convolution Convolution with user defined or pre-defined Kernels

- Scale free Matching Convolution, Statistical Maxima from Loops of Convolution with different image sizes.

Translation into Media processing operators

/ 4632

Evaluations

1. Design principles • Humans as sensors to detect real world events

2. Data representation and Situation recognition algebra

• Expressive, computable and explicit• Real world results

3. Framework for situation recognition • modeling, • situation evaluation, • personalized alerts

/ 4633

Humans as sensors• Can social media be used to detect real world events?

S.No Category Event Physical DateObserved

Temporal PeakPhysical Location

Observed Spatial Peak

1 Politics Health Care Bill passed 2010-03-21 2010-03-2138.89, -77.03 (Washington)

41, -74

2 Politics California Prop 8, Trial Day 1 2010-01-11 2010-01-1137.77, -122.41

(San Francisco)38,-122

3 Society Fort Hood Shootings 2009-11-05 2009-11-0531.13, -97.78

(Fort Hood, TX)33,-97

4 Society SeaWorld Whale Accident 2010-02-12 2010-02-1228.54, -81.38(Orlando, FL)

29,-81

5 SportsWinter Olympics Opening ceremony

2010-02-12 2010-02-1249.24, -123.11(Vancouver)

44,-79

6 Sports Baseball World Series final 2009-11-04 2009-11-0440.71, -74.00(New York)

41, -74

7 Entertainment Oscars 2010-03-07 2010-03-0734.05, -118.24(Los Angeles)

34, -118

8 Entertainment South by Southwest festival2010-03-12 to

2010-03-212010-03-15

30.26, -97.74(Austin, TX)

30, -98

9 Tech. Conv. CES 20102010-01-05 to

2010-01-072010-01-06

36.17, -115.13(Las Vegas)

34,-118

10 Tech. Conv. TED 20102010-02-10 to

2010-02-132010-01-10

33.76, -118.19(Long Beach,CA)

34, -118

/ 46

Data representation + Algebra

• Applications • Business analytics • Political event analytics• Seasonal characteristics

• Data • Twitter feeds archive

• Loops of location based queries for different terms• Over 100 million tweets using ‘Spritzer’/ ‘Gardenhose’ APIs

• Flickr feeds

• API: Tags, RGB values from >800K images

• Implementation • Matlab + Java + Python

34

/ 4635 AT&T retail locations

AT&T total catchment area

iPhone theme based e-mage,Jun 2

Aggregate interest

Under-served interest areas

-Subtract

DecisionBest Location is at

Geocode [39, -122] , just north of

Bay Area, CA

@Spatial.Max

<geoname><name>College City</name><lat>39.0057303</lat><lng>-122.0094129</lng><geonameId>5338600</geonameId><countryCode>US</countryCode><countryName>United States</countryName><fcl>P</fcl><fcode>PPL</fcode><fclName>city, village,...</fclName><fcodeName>populated place</fcodeName><population/><distance>1.0332</distance></geoname>

+ Add

to Jun 15, 2009

Convolution.

*Store

catchment area

Convolution .

*Store catchment

area

/ 4636

Seasonal characteristics analysis• Fall colors in New England

• Show me the difference between red and green colors for New England region, as it varies throughout the year.

• subtract(@spatial(sum)(πspatial(R=[(40,-76), (44,-71)]) (TEStheme=Red)), @spatial(sum)

(πspatial(R=[(40,-76), (44,-71)])(TEStheme=Green)))

Jan

0

Dec

/ 46

Building applications using the framework

S.No Application Data UsedApplication deployed?

ScaleData

modalitiesOperators

used

1Wildfire detection in California

Real Yes MacroSatellite data,

Google insightsF, A, Ch

2 Hurricane monitoring Simulated No Macro n/a F, A, Ch, P

3Flu epidemic surveillance

Real No Macro Twitter, Census F, A, C

4Allergy/ Asthma recommendation

Real In-progressMacro,

Personalized alerts

Twitter, Air Quality, Pollen

CountF, A, C

5Thailand flood mitigation

Real YesMacro,

Personalized alerts

KML F, A, C

Legend:F = Filter, A = Aggregate,C = Classification, Ch = Characterization, P = Pattern Matching

37

/ 46

Wildfire recognition model (Satellite data)

Absolute value variation

Spatial Neighbor variation

Fire detector(Satellite driven)

Significant band variation?

Hot enough?

∏

∏∏

Emage (Mid IR surface temp.)

Emage (4 µm temperature)

Difference value

Emage (11µm temperature)

Spatial Neighborhood Difference

Difference value

Neighborhood Mean value

Difference value

Unclouded?

∏

Emage (12 µm band temp.)

Δ Δ

SatelliteBand 4

AND

ANDThresh=310

Thresh=392

Thresh= 5Thresh= 30

Subtract Subtract

Convolve (7X7)

ϵ {fire, non-fire},<California, 24hrs, 0.01x

0.01>

SatelliteBand 12

LAADS.com,<California, 24hrs,

0.01x 0.01>

LAADS.com,<California, 24hrs, 0.01x

0.01>

SatelliteBand 12

LAADS.com,<California, 24hrs, 0.01x

0.01>

SatelliteBand 12

LAADS.com,<California, 24hrs, 0.01x

0.01>

38

/ 46

Wildfire recognition model (Social data)

Spatially anomalous Temporally anomalous

Fire detector(Social)

∏∏

Difference with other areas today

Emage (Google Insights- Fire)

Δ

Google Insights-Fire

Difference with Historical average

And

Thresh= 5 Thresh=7

Subtract

Spatial Avg. of Interest

Emage (Google Insights- Fire)

Δ

Google Insights-Fire

Average

Emage (Google Insights- Fire)

Δ

Google Insights-Fire

Emage (Google Insights- Historical Avg)

Δ

Google Insights-Fire

Subtract

ϵ {fire, non-fire},<California, 24hrs,

0.01x 0.01>

Google.com/insights,<California, 24hrs,

Metros>

Google.com/insights,<California, 24hrs,

Metros>

Google.com/insights,<California, 24hrs,

Metros>

Google.com/insights,<California, 24hrs,

Metros>

39

/ 46

Wildfire recognition

Fire detector (Social)

Fire detector (Satellite)

Fire detector

OR

ϵ {fire, non-fire},<California, 24hrs,

0.01x 0.01>

2010 2011 Total0

5

10

15

20

25

30

35

40

45

50

Social detectorSatellite detector Combined Ground truth

Number of Wildfires detected

Situation Modeling

Situation Evaluation

Results

40

/ 4641

Demo: Asthma Recommendation Application

/ 4642

Thailand Flood mitigation

/ 4643

Social Life Networks

43

Connecting People

Aggregation and

Composition

Situation Detection Alerts

Queries

Information

Situation aware routing

and Resources

Jain, Singh, Gao: Social Life Networks for the Middle of the Pyramid, ACM Workshop on Social Media Engagement ‘11.

/ 4644

Related Work: SnapshotArea Combine

hetero data

Human sensors

Data analytics

Define situations

Location aware

Real-time streams

Toolkits

Situation Awareness

X X o o X

Situation Calculus

X

Web data mining

o X X o X

Social media mining

o X X o X

Multimedia Event detection

X o o o

Complex event processing/Active DB

X X o X

GIS X o X X o

Mashup toolkits(Y! pipes, ifttt)

X X o X X

X

X X

X

X

X X

X

X

X

X

X

This work X X X X X X X

XX

o = partial support

/ 4645

Future work

• EventShop: • Personalization• Scalability• Prediction

• Using such tools to nudge people into taking desired actions

• Supporting Grids and Graphs for analysis• Social Life Networks

/ 4646

Summary

• Personalized Actionable Situations • 1st Systematic approach• Situation Modeling • EventShop: Web based system for Situation Evaluation

• Apps: Democratize data and action taking • Eco-system for data-to-action

/ 46

THANKS !

/ 46

BACKUP SLIDES

48

/ 4649

Analyzing Big DataField/ Approach Databases Networks Spatio-temporal

Data structure Tables Graphs Grids

Apps Business records, Banking

Internet traffic, Social network, Roads

Healthcare, Disaster relief, Business, Security

Problems Querying, Searching Shortest path, influence, anomaly

Situation detection

Operators Select, Project, Join Diameter, influence detection, connected components

Select, Aggregate, ST characterization, ST pattern matching, Classification

Modeling ER modeling, Query plan

Network diagrams, PetriNets

Situation models

Tools SQL server, Oracle DBMS

NS2, NetworkX EventShop

Situation detection

/ 4650

Geo-Social Power Laws• Studied 5.6 Million Tweets for a month• There is a fixed relative ratio for the occurrence of events

of different magnitude across space or time.

Whole world

AroundNew York

city

Only USA

Log(Rank)

Log(Magnitude)

Across Space

1 week

30 mins

1 day

2 weeks

1 month

3 weeks

Log(Rank)Log(Magnitude)

Across Time

Singh, Jain: Structural Analysis of Emerging Event-Web, (Short Paper) World Wide Web Conference‘10.

/ 4651

Situation Modeling

• A conceptual step before physically implementing situation detection filters • Analogy: E/R modeling, UML

• Helps domains experts externalize concepts e.g. ‘Epidemic’

/ 46

Data Supporting parameter(s) OutputOperator Type

Building Blocks: Operators

Δ Transform …Spatio-temporal

window

Φ Learn Learning method

{Features}

{Classification}

w w = {0.3, 0.6, 0.1}

52

Aggregate +

ClassificationClassification

method

@ Characterization Growth Rate = 125%

Property required

Pattern Matching72%+

Pattern

∏ Filter +Mask

/ 46

Queries• Seasonal characteristics

• Show me the segments based on average greenery, as they vary over the year.

• kmeans(n=3)(∏temporal(t>1293840)(TEStheme=‘green’))

• Political event analytics• Show me the difference of

interests in Personalities (p1, p2) in places where H is an issue.

• mult(diff(TEStheme=p1,TEStheme=p2), thresholds(30)

(TEStheme=H))p1=Obama, p2=Romney, H=Guns,

Aug 9, 2012, via EventShop

53

/ 46

Modeling personalized situations

54 /

Personal threat level c ϵ {Low, mid,

high}

Physical exertion

Asthma threat level

TPS (Funf)

Δ

Funf-activity

Phone sensors, (relaxMinder app),

[USA, 6 hrs,0.1x 0.1]

EventShop

∏Normalize (0, 100)

And

Classification: Thresh(30,70)

∏ Normalize (0, 100)

[USA, 6 hrs,0.1x 0.1]

TPS (Asthma)

∏UserLoc

/ 4655

Asthma Recommendation Application

Asthma Threat level

Allergy reportsPollen Count

∏

Emage (Pollen Level)

Δ

Visual-Pollen level

Air Quality

∏

Emage (AQI.)

Δ

Visual-Air quality

∏

Emage (Number of reports)

Δ

Twitter-Allergy

c ϵ {Low, mid, high},[USA, 6 hrs,

0.1x 0.1]

Weather.com,[USA, 6 hrs,

0.1x 0.1]

Twitter API,[USA, 6 hrs,

0.1x 0.1]

Pollen.com,[USA, 6 hrs,

0.1x 0.1]

Macro situation model

/ 46

Asthma threat: personalized situation

Personal threat level c ϵ {Low,

mid, high}

Physical exertion

Asthma threat level

TPS (Funf)

Δ

Funf-activity

Phone sensors, (relaxMinder app),

[USA, 6 hrs,0.1x 0.1]

EventShop

∏Normalize (0, 100)

And

Classification: Thresh(30,70)

∏ Normalize (0, 100)

[USA, 6 hrs,0.1x 0.1]

TPS (Asthma)

∏UserLoc

56

/ 4604/10/2023 Proprietary and Confidential, Not For Distribution 57

iPhone: Interest over 12 days.

/ 4658

S4) Situation detection operators