situation based analysis and control for supporting event-web applications
DESCRIPTION
Situation based analysis and control for supporting Event-web applicationsTRANSCRIPT
Situation based analysis and control for supporting Event-web applicationsVivek Singh Advisor: Professor Ramesh Jain
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Research Plan
Event-web
•Events and objects as basic organization and linking mechanism▫Multimodal▫Closer to real world
•Users gain insights and experiences
Events everywhere– (near future)•Events are all around us.
▫Ubiquitous sensors▫Excellent signal processing
techniques▫Wide-spread information broadcast▫Excellent data management
techniques•Large volumes of event data,
streaming in real time.•How can we use it? – machines
don’t understand them.
Motivation: From events to situations…•Given a plethora of event data. How can
we:▫Disambiguate relevant and irrelevant
events?▫Combine events into meaningful
representations ?▫Allow inference and cascading effects▫Support different interpretations based on
application domain▫Support Control & decision making
Event based reasoning
Symbolic inference
Domain semantic
sControl
Situation based control: Motivations
1. Inherent support for event-based (temporal) reasoning
2. The ability of the controller to reason based on symbols (rather than just signals)
3. Explicit inclusion of domain semantics (to support multiple applications)
Related WorkArea Sample
workEvent-based
Symbolic inferenc
e
Explicit Domain
semantics inclusion
Decision
making
Focus area
Situation Awareness
Endsley98 X X Defense/ Tactical
Situation Modeling
Yan06 * X Databases
Situation Management
Jakobson07 * X X Defense/ Tactical
Situational Control
Pospelov86 X Semiotics/ Linguistics
Event detection
Jain03 X Vision/ Multimedia
Knowledge based systems
Sullivan86 X X X Intelligent systems
Discrete Event Control
Ho89 X X Control theory
Situation Calculus
McCarthy69
X X * Logic
Situation based control
This work X X X X Symbolic Control
Applications
•Energy efficient buildings:▫When to switch off air-conditioner?
•Telepresence:▫Which camera feed to send out?
•Business analysis:▫What should be the correct price for iPhone?
•Earthquake rescue effort:▫Where to send out the next fire-fighter
engine?
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control ▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Research Plan
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Research Plan
E2E communication: Project Overview
Sentient Information
System
Sentient Information
System
Environment 1
Environment 2
Web
Device to Device communication
Towards Environment to Environment (E2E) multimedia communication systems, in Multimedia Tools and Applications Journal, Springer Netherlands, 2009.
Also in: ACM Workshop on Semantic Ambient Media Experiences (SAME), ACM Multimedia workshop, 2008.
Env. 1
Env. 4
JSM 2
JSM 1
Env. 5Env. 3
Env. 2
Joint SM
Shared Visualization Spaces for Environment to Environment Communication , in Workshop on Media, Arts, Science and Technology (MAST 09), 2009.
E2
E
Com
mu
nic
ati
on
Natural interactio
n
Semantic interactio
n
Seamlessinteractio
n
Bi-directional connectivit
y
Not depend on physical similarity
Handle privacy
Event-based architectur
e
Scalable architectur
e
Sensor abstraction
Multimodal information
No fixed application
Live and archived modes
Network/
Transmission
Environment Model
Environment Server
Situation based
controller
Actuator / Presentatio
n Model
MMDB
Sensors
Actuators / Presentation Devices
Physical Environment
EventBase
Environment: Node Architecture
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Research Plan
Situation Calculus: Quick overview▫ enter(P1), startWork(P1)▫ enter(P1), exit(P1), enter(P1), startWork(P1),
stopWork(P1), startWork(P1)
- isInRoom(P1, s(k))- isWorking(P1, s(k))
Events
Fluents
isInRoom(P1, s) ˄ ~isWorking(P1, s) → IncreaseMusicVolume() Control
isInRoom(P1, s) 0
isWorking(P1, s)
01
1 Situation
Situation = Not events , nor sequence of events, but their assimilated descriptor
Situation calculus: Basics (1/3)
•Logic formalism designed for representing and reasoning about dynamical domains.
•It builds upon traditional predicate, 1st and 2nd order calculus, but is different because it allows for truth values to change over time.
•Situation:▫“The set of necessary and sufficient world
state descriptors for undertaking control decision”.
Situation Calculus: Basics (2/3)•Ω = {A, S, O, F}
▫Actions (A) for actions i.e. those which change the 'state’ of the world. A= Aex U Asys
▫Situation (S) for `history of events' ,▫Objects (O) as the default sort for everything
else,▫Fluents (F) are predicates reified with situations.
(value assignments which change with time). Relational (give True/False answers) or Functional (return any value as computed)
•Do(action, situation): A X S → S
Situation Calculus: Basics (3/3)•D = Dfnd U Duna U ε U Dap U Dss U D0
▫Dap is a set of action precondition axioms, one per action symbol A.
▫Dss is a set of successor state axioms (SSAs), one for each fluent symbol f, which characterizes all the ways the value of a particular fluent can be changed.Poss(a, s) → [F(x, do(a, s)) ↔ γ+
F(x, a, s) ˅ ((F(x, s)˄ γ-
F (x, a, s))]
▫D0 is a set of axioms describing the initial situation S0.
Control theoretic problem formulation
• Aex(k) = f1(U(k))
• Inp(k) = f2(Aex(k) , Asys(k-1) )
• S(k) = f3(Inp(k) , S(k-1) )
•Asys(k) = f4(S(k), G)•Asys(k) = f4(f3(Inp(k) , S(k-1) ), G)•Y(k) = f5(Asys(k)
Implementing the controller
•Do(A, S) → S’.•S(k) = Do(Inp(k), S(k-1))
•Φ1(X1) →α1 (X1);• ...•ΦN(XN) →αN(XN);
•SGoal |= (~Φ1(X1) ˅α1(X1)) ˄ ... ˄(~ΦN(XN) ˅αN(XN))
• ᴲAsys(k) : Do(Asys(k), S(k)) → SGoal
S(k) = f3(Inp(k) , S(k-1) )Asys(k) = f4(S(k), G)
Implementing the controller
D’ = D U Dca
D’ = Dfnd U Duna U ε U Dap U Dss U D0 U Dca
Situation Based Controller
A. InferenceEngine
C. System Goal
B. Knowledge Base
Situation modeling
1. Identify the relevant Objects (O) , Actions (A) and Fluents (F)
2. Identify the preconditions for each action (Dap)
3. Identify the after-effects of each action (Dss)
4. Describe the initial situation (D0)
5. Identify the goal state using action-condition constraints (Dca)
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Research Plan
Loc 1: Desk Loc2: Whiteboard
Loc 3: Engineering
Model
Actions possible:1. Work on PC2. Work on Table
Conditions Actions
Move to location
Activity Selected
Cam
Desired Volume
Desk WorkOnPC
1 1
Desk WorkOnTable
2 2
Whiteboard
- 3 3
Model - 4 4User
Situation modeling: E2E application
Situation based control for cyber physical environments, Accepted: IEEE workshop on situation management, MILCOM, 2009
Step 1: Identify the relevant Objects, Actions and Fluents.
Step 2: Identify the preconditions for each action
Step 3: Identify the after-effects of each action
Step 4: Describe the initial situation
Step 5: Identify the goal-state using the action-condition constraints
Finding the optimal control action
Sample executions
DecreaseVolume, DecreaseVolume, DecreaseVolume, S0
•Exogenous action: MoveToLoc(`Model’) at the end of second cycle
IncreaseVolume, IncreaseVolume, SelectCam(4) MoveToLoc(`Model’), DecreaseVolume, DecreaseVolume, S0
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Research Plan
Research Challenge 1: Generic adaptability•Tools to allow system designers to
undertake their domain’s situation modeling •Necessary and sufficient details for handling
application•Discrete, hybrid or continuous•Current status:
▫Dap U Dss U D0 U Dca
•To Do▫Providing easy tools for users to inscribe such
domain knowledge
Research Challenge 2: Enhanced sensing based on feedback •Top down+ bottom up sensing
▫Sensing = F(current_state)•Detect and discard noisy event data.
▫Only allow valid sequences of input events▫Invalid(Seq) ↔(KB U S0 |= ¬Seq)
▫Discard (WearSocks >(T) WearShoes)
•Anomaly detection using these techniques▫Event based (semantic) level not signal
level
Research Challenge 3: Reasoning and analysis•Minimal representation: Find the minimal
set of events Emin which lead the situation changing from S0 to SGoal.
•Handling un-observable systems:▫Can we find the unknown state S0, by looking
at patterns of events and the changes in the system state (fluents) [e.g. in Chess]
•Approach:▫Using planning and projection operators of
situation Calculus
Research Challenge 4: Using Predictive Analysis for control action•Using estimates of future exogenous
actions for better control •Signal based data
▫Kalman Filter▫Model Predictive Control
•Symbolic data▫Semantic Kalman filter?
“Coopetitive multi-camera surveillance using Model Predictive Control”, Machine Vision and Applications Journal, 2008.
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Research Plan
Situ-itter: Looking beyond rooms…
•Can an entire city or country be considered a cyber physical system.
•Humans as sensors:▫Everywhere !▫Perception, Censors, Rumors, Delays
•Applications▫Should iPhone price be increased/decreased?▫Detect swine flu in Mexico ->> Issue pork-
import health warnings in Alaska▫DEMO
Research Challenge 5: Scalability of situation based control •Number of Events and conditions to be
considered▫Hierarchical approach
•Supporting multiple applications with different complexity levels▫Creating models for different applications
•Approaches:▫Allow users to define models ▫Learn patterns ▫Use public knowledge/ Ontologies
Outline
A. BackgroundB. E2E project
▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges
C. Situ-itter▫ Overview▫ Research challenges
D. Summary and Plan ahead
Current status: Systems • E2E project
▫Working prototypes DBH2059, CalIT2
▫Skype based lite-version▫Collaborative nodes
National university of Singapore (Observation System) INRIA, France (emotion enhanced E2E)
• Situ-itter▫Proof-of-concept
• Multimodal observation systems, ACM Multimedia 2008. • ObSys: A Generic Sensing Architecture for Multimodal Observation Systems, Submitted
to TOMCCAP: ACM Transactions on Multimedia Computing, Communications and Applications
• Toward Environment-to-Environment (E2E) Affective Sensitive Communication Systems, submitted to: MTDL workshop, ACM-MM, 2009.
Future work: Systems
•Robust bi-directional E2E communication between UCI, and Singapore
•Implementing situation controller into physical sensors
•Building Twitter crawler/ real-time analysis tool
Area Challenges Status Type of contribution (expected)
Approach
Overall Framework
Temporal + Symbolic reasoning
Prelim. Tools Situation Calculus
Use domain semantics
Prelim. Tools Situation Modeling
Generic & Scalable
Support Multiple applications
Prelim. /Plan
Tools -User tools-Learning -Ontologies
Large number of events
Plan Tools Hierarchical Control
Reasoning and Analysis
Minimal event set Plan Logic-based Min (Seq) : Do(Seq, S0) -> Sgoal
Partial Observability Plan Logic-based S0: Do(Seq, S0) -> Sgoal
Feedback enhanced sensing
Noisy event data , anomalies
Plan Logic-based Invalid (Seq)<-> KB U S0 |= ¬Seq
Top-down + bottom up sensing
Plan Optimality Sensing =F(S_curr)
Predictive Control
Sensor/ device selection
Plan Optimality Symbolic Kalman Filter+ Model Predictive Control
Research Plan
•In progressing order of importance for my work
•Year 3 --Tools▫Finalize overall framework ▫Make it generic and scalable
•Year 4 – Logic based approaches▫Use inference, reasoning and analysis▫Feedback enhanced sensing
•Year 5 – Optimality based contributions▫Predictive Control
Publications• E2E
1. {VKS, HP, IR, RJ}: Towards Environment to Environment (E2E) multimedia communication systems, in Multimedia Tools and Applications Journal, Springer Netherlands, 2009.
2. {VKS, HP, IR, RJ}: Also in: ACM Workshop on Semantic Ambient Media Experiences (SAME), ACM Multimedia workshop, 2008.
3. {VKS, IR, RJ}:User availability detection in E2E systems, in Workshop on Media, Arts, Science and Technology (MAST 09), 2009.
4. {HP, VKS, AM, RJ}: Shared Visualization Spaces for Environment to Environment Communication , in Workshop on Media, Arts, Science and Technology (MAST 09), 2009.
5. {IR, VKS, HP, RJ}: Environment to Environment (E2E) communication systems for collaborative work, Poster in Computer Supported Cooperative Work (CSCW) 2008.
VKS=Vivek Singh, HP=Hamed Pirsiavash, IR=Ish Rishabh, AM=Aditi Majumder, RJ=Ramesh Jain
Publications• Situation based control
1. {VKS, RJ}: Situation based control for cyber physical environments, Accepted: IEEE workshop on situation management, MILCOM, 2009
• With external collaborators1. {MS,VKS, RJ, MK}: Multimodal observation systems, ACM
Multimedia 2008. 2. {MP,VKS, BH,RJ}:“Toward Environment-to-Environment (E2E)
Affective Sensitive Communication Systems”, MTDL workshop, ACM-MM, 2009.
3. {MS,VKS, RJ, MK}: ObSys: A Generic Sensing Architecture for Multimodal Observation Systems, Submitted to TOMCCAP: ACM Transactions on Multimedia Computing, Communications and Applications
4. {VKS, RJ, MK}: Motivating contributors in Social media networks, submitted to: ACM MM workshop on Social media.VKS=Vivek Singh, RJ=Ramesh Jain, MS=Mukesh Saini,
MK=Mohan Kankanhalli, MP=Marco Paleari, BH=Benoit Huet
Publications• Prior work: Master’s thesis
1. “Coopetitive multi-camera surveillance using Model Predictive Control”. Journal of Machine Vision and Applications, 2009.
2. Adversary aware surveillance systems, IEEE TIFS, Trans. Info. Forensics and Security, 2009.
3. “Coopetitive Multimedia Surveillance”, International Conference on Multimedia Modeling (MMM'2007).
4. "Towards adversary aware surveillance systems", IEEE International Conference on Multimedia and Expo, (ICME-2007).
5. A Design Methodology for Selection and Placement of Sensors in Multimedia Surveillance Systems”, ACM Multimedia Workshop on Video Surveillance and Sensor Networks (ACM MM, workshop-VS SN 06)
6. “Coopetitive Visual Surveillance using Model Predictive Control”, (ACM-Multimedia, workshop-VSSN 05)
• Journals (3 accepted, 1 submitted), • Conferences (4), • ACM-MM workshops (5),• Other venues (3)