Design and Implementation of a Middleware for Sentient Spaces

Bijit Hore, Hojjat Jafarpour, Ramesh Jain, Shengyue Ji, Daniel MassaguerSharad Mehrotra, Nalini Venkatasubramanian,Utz Westermann

School of Information and Computer ScienceUniversity of California, Irvine

SATware: Middleware for Sentient Systems

2

Outline

Rescue project & Responsphere Multimodal sensor data processing SATware architecture

Stream model SATRuntime: Stream processing runtime SATLite: Stream query and transformation language SATDeployer: Operator deployment Privacy

SATware Demo Related work

SATware: Middleware for Sentient Systems

3

Rescue project and Responsphere Exploiting IT for effective response to unexpected events

Responsphere: a unique infrastructure for situation monitoring and awareness in the context of emergency response Includes more than 200 sensors of 10 different types deployed over the

geographical space that covers about half of the UCI campus.

SATware: Middleware for Sentient Systems

4

Multimodal sensor data processing: Application

SATware applications: Surveillance and monitoring Emergency response

Situation awareness Drills

SATware: Middleware for Sentient Systems

5

Surveillance Application

A building is equipped with multimodal sensors

Video cameras. RFID readers, Motes,..

Specific event should be detected and notifications should be generated

SATware: Middleware for Sentient Systems

6

Multimodal sensor data processing: Challenges Abstraction

Entities and spaces Virtual sensor concept

Multimodal stream processing Heterogeneous stream sources e.g., video cam, RFID, mote, ..

Scalability Extensibility Mobility Privacy

SATware: Middleware for Sentient Systems

7

SATware

A distributed middleware for capturing semantically meaningful events

Provides higher level of semantic abstraction compared to sensor middleware

Two level stream processing: Semantic event stream processing Sensor data stream processing

SATware: Middleware for Sentient Systems

8

QoS Privacy

Stream

Stream

Stream

StreamStream

Stream

QoS Privacy

Situationmodel

Domainmodel

Uninterpreted streams of sensor event observations

Interpreted streams of entity-related events

Sensing layer

Semantic layer:entities and activities

User

Application

SATware stream processing model

SATware: Middleware for Sentient Systems

9

SATware architecture: Stream model Data model

A stream is a flow of records <t , C , Confidence> t : time stamp C: content Confidence: probability of the correctness of tuple content

< 12:10:25 , PersonID = 7 , 0.9 > < 11:32:31 , RoomOccupancy = 4 , 0.75 >

SATware: Middleware for Sentient Systems

10

SATware architecture: Stream model Processing model

Operators Input, output buffer

Operator types Image processing operators Join operators Synchronization operators Filtering operators …

Oi

SATware: Middleware for Sentient Systems

11

SATware architecture: Stream model

Oi

< ti , PersonID >

< ti, Door Status>

< ti , PersonEntery >

Oi

< ti , Jpeg image > < ti , Door Status >

Sample image processing operator

Sample join operator:

Oi

< ti , PersonID >

< ti, Door Status>

< ti , PersonID , Door Status>

Sample synchronizer operator:

SATware: Middleware for Sentient Systems

12

SATware architecture

SATware: Middleware for Sentient Systems

13

SATware architecture: SATRuntime Main components

Set of processing nodes SATRuntime component

One SATRuntime on each processing node Interface for operators (running, accepting, migrating,

connecting,…) Operators as mobile agents

Input stream, output stream buffers Operator connection interface Operator migration

SATware: Middleware for Sentient Systems

14

SATware architecture: SATRuntime

SATRuntime infrastructure directory Available streams Sensor and Operator repository

Sensor management public boolean addSensorInstance(SensorInstance sensorInstance); Sensor.setLocation( Location l );

Operator management public boolean addOperatorInstance(OperatorInstance operatorInstance);

Deployment status Operator graph Operator location

SATware: Middleware for Sentient Systems

15

SATware architecture: SATLite SATLite: Defines the query plan

Set of interconnected operators fed by sensors SATLite provides a query plan by translating the query

into a graph Select t , PersonID from DoorStatus and PersonID where Person.t

= DoorStatus.t and DoorStatus=‘OPEN’

Oi

< ti , PersonID >

< ti, Door Status>

< ti , PersonID , Door Status>

Oj

< ti , PersonID >

SATware: Middleware for Sentient Systems

16

SATware architecture: SATDeployer Translating a query plan into a deployment

plan Mapping query plan nodes into processing nodes

Minimizing query processing cost Network cost Computing cost Query evaluation time

Optimizations Operator reuse Load balancing

SATware: Middleware for Sentient Systems

17

SATware architecture: Privacy Hiding individual’s identity

Revealing individual’s identity when a predefined policy is violated

Sensors are trusted components

Adversary is an inquisitive but non-interfering (passive) entity that can observe all stored data and intercept all communication between two components of the system.

SATware: Middleware for Sentient Systems

18

Sentient building demo

A pervasive space equipped with sensors Video cams, RFID readers, Motes, Mobile sensing

platforms,…

SATRecorder: a building surveillance application

Recording video feed based on timed schedule or event detection

Coffee room scenario Sensors: 2 video cams, RFID reader Sample policy: if pot is empty and burner is on record

video

SATware: Middleware for Sentient Systems

19

Sentient building demo

SATware: Middleware for Sentient Systems

20

Sentient building demo

SATware: Middleware for Sentient Systems

21

Sentient building demo: coffee room Event definition:

Select PersonID from KitchenEvent where CoffeeLevel = ‘EMPTY’ and PotInPlace = TRUE and BurnerStatue = ‘ON’

Network model and query plan

SATware: Middleware for Sentient Systems

22

Sentient building demo: coffee room

SATware: Middleware for Sentient Systems

23

Related work

Data stream processing Aurora, STREAM, TelegraphCQ

Sensor networks Multimedia stream processing

IBM S3, MedsMan, IrisNet, Smart Camera Network

SATware: Middleware for Sentient Systems

24

Conclusion

We proposed: Challenges in sensor stream processing SATware: A middleware for sentient spaces A surveillance application based on SATware

Future plan: Expanding sensor deployment Sophisticated event detection Improving SATLite and SATDeployer Integrating more sophisticated privacy preserving

techniques