from sensor networks to smart environments and social networks

From Sensor Networks From Sensor Networks to Smart Environments to Smart Environments

and Social Networksand Social Networks

From Sensor Networks From Sensor Networks to Smart Environments to Smart Environments

and Social Networksand Social NetworksHamid AghajanHamid AghajanAmbient Intelligence Research LabAmbient Intelligence Research LabStanford University, USAStanford University, USA

and Social Networksand Social Networksand Social Networksand Social Networks

Ambient Intelligence Hamid Aghajan

IntroductionOutlineOutline

Introductiono Trends in technology and researcho From Smart Environments to Ambient Intelligenceo New potentials in user-centric, context-aware systems

Our labo User-centric ambient intelligence applications

Human activity analysisHuman activity analysiso Source of context in smart environments

Adaptive smart homes Well-being applicationsAdaptive smart homeso Learning from user feedback

Well being applicationso Avatars and social interactions

Meetings of the future Environment discoveryMeetings of the futureo User-centric performance evaluation

Environment discoveryo User interactions as contextual clues

Vision: Application LandscapeVision: Application LandscapeVision: Application LandscapeVision: Application Landscape


Application LandscapeApplication LandscapeVision offers rich Vision offers rich contextualcontextual data:data:

3D modeling

Applications established in monitoring and surveillanceApplications established in monitoring and surveillanceUserUser--centric, homecentric, home--based application market ? based application market ? Need value propositionNeed value proposition

modeling

Urban sensing

GamingAvatars

Intelligent vehicles

Face profile: tele-presence, remote gaming


Ambient IntelligenceAmbient IntelligenceRetail: interactive ads Ambience controlSeminar rooms

Intelligent light control 3D tele-presence Assisted living


Ambient IntelligenceAmbient IntelligenceA li ti i i t d li iApplications in assisted living

Event and scene descriptionEvent and scene description

Monitoring patient events

Key to adoption:Key to adoption:Reliability, ease of use, pReliability, ease of use, privacy management, data security, user control (opt in/out)rivacy management, data security, user control (opt in/out)

Research & Development TrendsResearch & Development TrendsResearch & Development Trends Research & Development Trends Personalization of ServicesPersonalization of Services


UserUser--centric application spacecentric application spaceTrendsTrends

pp ppp p

Smart Environments

Ambient Intelligence

user adaptation

behavior models gAdaptive, unobtrusive, Intuitive,context-aware

Sense, Interact,

perceive, interpret,

behavior models

UserContext

Social

services tailored to learned user preferences

pproject,

react, anticipate

InteractionsMulti-modal, pervasive

connectivity / tele-presencefor sharing an experience g pwith others in a flexible,

layered visual representation


TrendsTrends

Smart Environments

Ambient Intelligence

user adaptation

behavior models gbehavior models

UserContext

Social

Interdisciplinary field of research:Interdisciplinary field of research:• Engineering

Interactions

Engineering• Multimodal sensing, distributed processing, networking, reasoning, interface design• Ambient communication, media convergence• User profile, behavior model, preferences, context

• Psychology:• Psychology:• Human factors, user activity, emotions, history, skills, limitations, sensitivities, privacy options

• Sociology:• Social networks, multi-user interactions, large-scale databases, behavior patterns


Pervasive sensing processing and communicationTrendsTrends

Pervasive sensing, processing, and communication (widespread technology support)

Personal devices: smart phones, laptops, smart cars, personal multimedia and gaming consoles (constant interface with the user)

ConvergenceConvergence of sensor nets media and virtual data domains:of sensor nets media and virtual data domains:ConvergenceConvergence of sensor nets, media, and virtual data domains:of sensor nets, media, and virtual data domains:Mobile device as interface of user’s physical world with digital worldAlso as carrier of user profile

o Physical world: sensors, user context, activities, interactions, events, home appliances

o Digital world: Media, TV, internet, search, home automation, g , , , , ,outdoor services, virtual

oo User profileUser profile: Store and update user behavior model, habits, : Store and update user behavior model, habits, preferences in different contexts for service adaptationpreferences in different contexts for service adaptationp pp p


TrendsTrendsUser • Behavior model, profile, preferences

Context-Aware

Modeling

• Inference of tasks and intentions

, p , p• Ambience control and smart home services• Social networks, pervasive communication

FusionFusionddrrrr

High-Level

Co e a eProcessing

• Human activity recognition (semantics)

Inference of tasks and intentions• User context for service provision, interrupts• Semantic labeling of things and effects

andandSemanticsSemantics

ehav

ioeh

avio

ehav

ioeh

avio

• Human pose analysis (graphical models, ti l filt h t f t )

High Level Reasoning

• Human activity recognition (semantics)• Object recognition with user interactionsSignalSignal

and and FusionFusionl l t

o B

eto

Be

l l to

Be

to B

e

Multi-Camera Vision

particle filters, heterogeneous features)• Face angle estimation, face profiles• Decision fusion for event detection

S tS tSign

alSi

gnal

Sign

alSi

gnal

Smart Camera Networks

• Occupancy counting• 3D reconstruction of events• Distributed embedded processing for real-

time human pose analysis

SystemSystemand and

SignalSignal

From

SFr

om S

From

SFr

om S

Computer Vision

Wireless Sensor

NetworksSample Methods Sample Methods and Applicationsand ApplicationsSample Methods Sample Methods and Applicationsand Applications

FFFF


TrendsTrendsSmartSmart S i t t i i t t j t t ti i t

Occupancy sensing and services

Smart Smart EnvironmentsEnvironments

Sense, interact, perceive, interpret, project, react, anticipate

– Provide services based on location, event type, number of occupants

• Smart lighting for energy efficiency / Ambient lighting• Conference room analytics, smart meetings and presentations

– Smart buildings / occupancy maps:• Real-time maps for emergency management and rescue

guidance

• History data for resource utilization: space, lighting, walkways, energy, activity patterns


TrendsTrendsSmartSmart S i t t i i t t j t t ti i t

Human activity

Smart Smart EnvironmentsEnvironments

Sense, interact, perceive, interpret, project, react, anticipate

– Analyze and react to pose-basedevents:

• Fall detection, gaming, HCI, gesture control, home automation, pervasive communication, smart presentations

– Reconstruct actions, expressions, 3D models, avatars

– Semantic labeling based on user interaction• User activity as context to discover the environment

Time


TrendsTrends

Ambient communication

A Vision for Future: Novel Application DomainsA Vision for Future: Novel Application Domains

Ambient communicationTele-presence, pervasive communication, avatars, social networks

Well-being applicationsActivity monitoring, exercise, daily routines, patient and elderly care

Energy efficiencyOccupancy based context aware adaptive to personal modelOccupancy-based, context-aware, adaptive to personal model

MultimediaContext-aware ambience control and games, sensor-enabled social networks g ,

Smart meetingsConference room analytics, adaptive and flexible tele-conferences, speaker assistance systemsassistance systems


UserUser--centric Designcentric Design

S t & Al ithSystem & Algorithm

Application

Algorithm design often ignores:• Ease of installation / use • Interruptibility of the user• Unobtrusiveness• User preferences and sensitivities

p y• User engagement level• User skills / limitations

• Privacy issues • User perception of control


S t & Al ith • Real time vision and visualization

UserUser--centric Designcentric DesignSystem & Algorithm • Real-time vision and visualization

• 3D reconstruction of action• Appearance-based or avatar• Avatar repertoire database• Action mapping, expression mapping

B h i d l M d f t ti

User acceptance

Social aspectsUser preferences• Behavior models• Context• Option to override

• Modes of representation• Privacy issues

• User-friendly installation and use • Context-aware operation and responseAttributes of user-centric design:

Application

User friendly installation and use

• Self configuration / automated environment discovery

• Learning behavior models for the user

Context aware operation and response

• Privacy options in multi-user communication applications

• Adaptation to user defined preferences• Learning behavior models for the user and the environment

• Adaptation to user-defined preferences– Balance of automation and user query

Affective interfaces: Regard for user’s skills, limitations, personality characteristics, emotional state in services or interrupts, infer user’s intention


Privacy ManagementPrivacy ManagementMulti-layered privacy handling approach neededMulti layered privacy handling approach needed

Need policy, convincing trust metricOpt in/out, user-centricOwners of privacy options: the elderly, family, nursing facility, insurance, legislation

Multi-modal sensingU th t t i t

Owners of privacy options: the elderly, family, nursing facility, insurance, legislation

– Use other sensors to trigger cameras upon event– Activate voice communication first to check status– Image query only possible by authorized personnel

Smart cameras

– Raw video saved locally for post-event analysis

Turn video into text in normal user stateTransform person’s gesture into:

silhouette, avatar

Designing a Practical Vision SystemDesigning a Practical Vision SystemDesigning a Practical Vision SystemDesigning a Practical Vision System


Interfacing VisionInterfacing Vision

Observe → interpret → build up behavior models → react

Quantitative knowledge + Qualitative assessmentSensing Context

Processing Behavior Model

R i t tResponsiveness to events– Adapt services– Employ additional sensors– Send alerts

InteractivityB d t l ti i f i t t f

Real-time visiona necessity

– Based on gesture, location, region of interest of user



Vi i• Pose estimation, activity recognition• Face and gaze analysisVision

algorithms• Face and gaze analysis• Event reconstruction• Tracking, identification



Vi iVision algorithms

Camera Node

Task:T ki ?

Energy consumption?

Data aggregation?Distributed

Observations

OperationVision System:

Mono or stereo?Resolution?

Field-of-View?

Camera orientation?

Tracking?Counting?

Data Exchange:Type of data?Traffic load?

Multi-camera hardware & network Network topology?

Camera orientation?Placement?

Vision algorithm:Local vs. central processing

Application Requirements:Accuracy? Coverage?

Network Lifetime?

Which cameras sense?



Vi i• Pose estimation, activity recognition• Face and gaze analysisVision


Operation • Local processing and centralized processing• Communication bandwidth• Latency of real-time results• Resolution in image view and time

Multi-camera hardware & network

Resolution in image view and time• Temporal alignment (synchronization)• Camera view overlaps, data redundancies• Data exchange methods


Interfacing VisionInterfacing Visionbehavior models

High-level

behavior context events adaptation

high-levelreasoning

observation knowledgei t falgorithms

vision

user interface 3D model

validationg

accumulationi n t e r f a c e

pose / activity face / gaze

Vi i

Data fusion

• Pose estimation, activity recognition• Face and gaze analysisVision







High-level

• Behavior models and user preferences• Contextual data• Knowledge-base from historic dataalgorithms • Knowledge-base from historic data• Communication mode and user availability

• Relative confidence levels in space and time• Fusion based on hybrid features

Vi i

Data fusion

• Pose estimation, activity recognition• Face and gaze analysis

• Assignment of priorities and tasks

Vision algorithms

• Face and gaze analysis• Event reconstruction• Tracking, identification





Traditional VisionTraditional VisionGeneric features: Can become unavailable or un-interestingGeneric features: Can become unavailable or un interesting

Frame scope: Frames can have zero or misleading information value

Calibration: User-dependent function

Point-to-point accuracy metric: May not be relevant to end application

E i t d fi itiEnvironment definition: Manual, need to repeat upon movement

Fixed vision function: To avoid overloading the processor in real-time app.g p pp

Uniform value for data: Misleading data can bias results


VisionVision for Ambient Intelligencefor Ambient IntelligenceGeneric features: Can become unavailable or un-interestingGeneric features: Can become unavailable or un interesting

Opportunistic features, context-driven

Frame scope: Frames can have zero or misleading information valueFlexible fusion window, reliability models

Calibration: User and environment-dependent functionObservation-based methods, application needspp

Point-to-point accuracy metric: May not be relevant to end applicationApplication-driven metric

E i t d fi itiEnvironment definition: Manual, need to repeat upon movementObservations of user interactions

Fixed vision function: To avoid overloading the processor in real-time app.g p ppAlg. switching, Task assignment (active vision)

Uniform value for data: Misleading data can bias resultsConfidence level based on content and historyConfidence level based on content and history

Enabled by two-way interfacing of vision with higher processing layers


Vision and HighVision and High--Level ReasoningLevel Reasoning

Maintain knowledge base by accumulating historic dataMaintain knowledge base by accumulating historic dataGuide vision processing based on history

Compensate for imperfect vision processing output, enhance robustness

Assess relative value of information:Value of current estimates based on past interpretationsValue of low-level features for addressing a high-level taskValue of low-level features for addressing a high-level taskValue of observations made by different cameras

Task assignment to different cameras based on recent

Case study (later): Interfacing vision with inference engine for object recognition based on user interactions

results, context, and current observations


Models, Context, FeedbackModels, Context, FeedbackHierarchical processing structure:Hierarchical processing structure:

Vision processing has a higher dependency on the environment and placement of cameras, semantic reasoning is more generalEach module can be designed and modified separatelyg p yThe same high-level model can be used in different environments

behavior environment events adaptationhigh-level

ireasoningobservation validation

knowledgeaccumulation

c o n t e x t i n t e r f a c e

vision

motion pattern 3D modelpose / activity face / gaze

vision hardware & network


Context in Vision ProcessingContext in Vision ProcessingSpatiotemporal constraintsGeographic informationOther modalitiesMulti-camera networkCamera priors

EnvironmentalContext

StaticContext

Domain knowledgeDomain knowledgeHigh-level reasoning

Context Context

User-centric Context

Dynamic Context

User interfaceUser interfaceUser location and activityUser behavior modelUser preferencesSocial settinggRepresentation mode

K. Henricksen, J. Indulska, A. Rakotonirainy, “Modeling context information in pervasive computing systems”, in Proc. of the First Int. Conference on Pervasive Computing, 2002.


Models, Context, FeedbackModels, Context, Feedback

camera network model(geometry, topology, roles, priors, confidence levels)

environment discovery(objects, temporal/affine

usage relations)

user model(routines, preferences,

emotional state)

abnormal event detection(activities, routines, accident,

control zones, crowds)

behavior modelscontext

semantic-level context


i

context

reasoningobservation validation


c o n t e x t i n t e r f a c e signal-level

context

vision


context

system levelvision

hardware & network

system-levelcontext

Ambient IntelligenceAmbient IntelligenceAmbient IntelligenceAmbient IntelligenceCase Studies in UserCase Studies in User--centric Application Designcentric Application Design


UserUser--centric Contextcentric ContextUser behavior model– Inference of intention– Detection of abnormal events– Emotional state (affective computing)– Demographical information

f– User profiles in personal devices: • Preferences, skills, special needs, knowledge, expertise, limitations(mobile phones, laptops, cars, smart rooms, multimedia / game consoles)

How can this help?– Analyze / anticipate events using knowledge base

Ch ll– Challenges:– Method to summarize / abstract sensed input into behavior models– How to query and deduct from the model in real-time responsesq y p– Modeling / tracking change in user behavior– Dealing with ambiguity or uncertainty in captured user information


UserUser--centric Contextcentric ContextUser preferencesp– Feedback

• Explicit (system-directed confirmation)• Implicit (user-directed confirmation)

– Interrupts: Provide the right service / data at the right time (proactive )g ( )• System-directed initiative (novice user)• Mixed initiative (experienced user)

How can this help?– Turn generic settings into adaptive services

User’s perception of being in control of the system– User s perception of being in control of the system

– Challenges:– What is proper level & type of user query?p p yp q y– Being proactive while keeping user interruption at minimum


UserUser--centric Contextcentric ContextRepresentation modep– Communication channel

• Visual / non-visual options• Raw video, voxel, shadow, avatar

How can this help?– Offer multiple options to preserve user privacy in visual communication

Employ suitable vision processing functions– Employ suitable vision processing functions


Our LabOur LabWSNLWSNL:: WWirelessireless SSensorensor NNetworks Labetworks Lab Multi-camera algorithms:WSNLWSNL: : WWireless ireless SSensor ensor NNetworks Labetworks Lab

• Occupancy sensing• Face detection• Best view selection• Vision-based data and

decision fusion• Distributed processing

WSNL.Stanford.edu


A I RA I R Lab:Lab: AAmbientmbient IIntelligencentelligence RResearch Labesearch LabOur LabOur Lab

AIRlab.Stanford.eduA I RA I R Lab: Lab: AAmbient mbient IIntelligence ntelligence RResearch Labesearch Lab

AdaptiveAdaptive, , contextcontext--awareawareapplications:applications:

• Activity classification• Ambience control• User behavior model• User feedback, HCI,• Energy efficiency• Social networks• Well-being, connectedness


User centric application design:Ambient IntelligenceAmbient Intelligence

User-centric application design:Ambient communication

Tele-presence and modes of representationp p

Adaptive smart homesLearning from user feedback

E i itExercise monitorAvatars and real-time social interactions

Meetings of the future / Speaker assistanceg pUser-centric performance evaluation

Environment and object discoveryUser interactions as source of contextUser interactions as source of context

TeleTele--Presence, Human Pose AnalysisPresence, Human Pose AnalysisTeleTele Presence, Human Pose AnalysisPresence, Human Pose AnalysisAvatars, Modes of RepresentationAvatars, Modes of Representation


M lti 3D t ti V l

Ambient Communication / TeleAmbient Communication / Tele--PresencePresenceMulti-camera 3D reconstruction - Voxels

Tommi Maataa

http://wsnl.stanford.edu/videos/examples/visualhull_cmp.avi

http://wsnl.stanford.edu/videos/examples/voxel_skin.mpg

WSNL - Stanford40


Ambient Communication / TeleAmbient Communication / Tele--PresencePresenceCustomized visualization:– User availability mode + avatar preferences:

B t i ( id )• Best camera view (video)• 3D shadow• Avatar replicating user’s gesture• Avatar with pre-defined gesture

– Change the 3D view based on viewer’s position

Applications in home-to-home communication and remote meetings

T. Määttä, A. Härmä, and H. Aghajan, “Home-to-Tome Communication Using 3D Shadows“, Immerscom 2009.


Multi-camera pose analysis - Avatars

Human Pose AnalysisHuman Pose AnalysisMulti-camera pose analysis - Avatars

http://wsnl.stanford.edu/videos/gesture/combine1.avi

http://wsnl.stanford.edu/videos/gesture/rotate2.avi http://wsnl.stanford.edu/videos/gesture/jogging1.avi

http://wsnl.stanford.edu/videos/gesture/combine1.avi

WSNL - Stanfordhttp://wsnl.stanford.edu/videos/gesture/pang.avi


Multi-camera pose analysis - Avatars

Human Pose AnalysisHuman Pose AnalysisMulti-camera pose analysis - Avatars

http://wsnl.stanford.edu/videos/gesture/realtime1.avi

43

p g

C. Wu and H. Aghajan, “Real-Time Human Pose Estimation: A Case Study in Algorithm Design for Smart Camera Networks“, Proceedings of the IEEE, Nov. 2008.


A user-centric system needs to support

Ambient Communication / TeleAmbient Communication / Tele--PresencePresenceA user centric system needs to support

a variety of visualization modescommunication

best view selection

3D reconstruction

avatarmapping textual report

visualization

c o n t e x t i n t e r f a c e

observation validation

user preferences and availability

3D modelmotion patternface / gazepose / activity

vision

vision hardware & networkSwitching between different vision

algorithms based on user’s choice

WellWell--Being ApplicationsBeing ApplicationsWellWell Being ApplicationsBeing ApplicationsAvatars, Social Networks, Health ReportsAvatars, Social Networks, Health Reports


WellWell--Being ApplicationsBeing ApplicationsSmart Homes: Health and wellSmart Homes: Health and well--beingbeing

Input from heterogeneous sensorsEvaluation: query the user, confirm with an expert, demographic comparisons

Smart Homes: Health and wellSmart Homes: Health and well--beingbeing

comparisonsMonitoring patient events


Exercise at Home?Exercise at Home?WellWell--Being ApplicationsBeing Applications

Exercise at Home?Exercise at Home?Benefits:– Comfort and privacy superior to gym– Comfort and privacy superior to gym– Flexibility in time User preferences in the

avatar domain

Disadvantages:– Absence of a personal coach or trainer– Lack of social motivation factors

(social atmosphere, competition)

Staying with exercise plan more likely with:likely with:– Measurements and record keeping– Social connectedness


Home Exercise MonitorHome Exercise MonitorWhat technology can offer– Instant measurements, instructions, and feedback

(Vision and HCI)– Interactive links with trainer

(C i ti d I t t)(Communication and Internet)– Different visualization options

(Computer Graphics)(Computer Graphics)– Virtual games with peers

(Social Networks)( )– User history and knowledge base

(Databases and Artificial Intelligence)

Also use of Psychology to assess user acceptance


Home Exercise MonitorHome Exercise MonitorWhat can this technology do?What can this technology do?– An exercise system at home in front of a TV equipped with a camera– The camera will project user’s image on the TV screen while you exercise

The image can be presented to the user in three ways: – Mirror image– “Body shape” avatar– Skeletal “stick figure” avatar

I b h d ith thImage can be shared with others– Fitness coach– Exercise buddies and friends– Other users in social network

The fitness coach could provide instructions, guidance, and corrective f db kfeedback– User may also be able to see the exercising movements of a coach, friends, and

other users through this system


Home Exercise MonitorHome Exercise Monitor

Requirements for acceptance:Requirements for acceptance:

– Real-time gesture recognition• Adaptation to user silhouette set

– Social connectedness• Live links with coach, peers

– User preferencesp• Instructions and feedback• Options on avatars


Application concepts:Exercise MonitorExercise Monitor

Application concepts:• Monitor, measure home exercise

– Instant feedback, instructions, progress report• Connection with a coach• Connection with a coach• Social networking

– Shared experience with peers via avatars

User preferences • Instructions and feedback

Instructions

Measurements

Progress report

Coach

FeedbackAvatar Social

Network

i t ti i tiinteractive communicative• Social connectedness• Group games


Home Exercise MonitorHome Exercise MonitorExercise MonitorExercise Monitor

Home Exercise MonitorHome Exercise Monitor

Use of context in silhouette search:• Prior knowledge of the exercise routine

Observed user silhouettes

• Prior knowledge of the exercise routine• Pose in previous frames

Reduce size of search regionsilhouettes

L b l d ( d b d j i t )

• Update silhouette bank with acquired user’s silhouettes

Adapt to user’s appearance over timeLabeled (pose and body joints) generic silhouette set

J. Cui, Y. Aghajan, J. Lacroix, A. van Halteren, H. Aghajan, “Exercising at home: Real-time interaction and experience sharing using avatars”, Journal of Entertainment Computing, Dec. 2009.


Exercise MonitorExercise Monitor

http://wsnl.stanford.edu/videos/exercise/Record5.avi http://wsnl.stanford.edu/videos/exercise/Record2av.avi

Human Activity LabelingHuman Activity LabelingHuman Activity LabelingHuman Activity LabelingSource of Context for Smart Home ServicesSource of Context for Smart Home Services


Human ActivityHuman Activity

Home environment is complex for activity analysis with visionanalysis with vision– Previous work usually have clean background or

well-positioned cameras

Fine-level activity classification is needed for ambient intelligence– Activities in the living room, kitchen, and study

roomroom


Human ActivityHuman ActivityCan serve as context for providing services in smart homes

0m

Philips Research

Home Lab

p g4.

30

9.42m

5.12

m

http://wsnl.stanford.edu/videos/envdisc/views_DiningTable1_T.avi

http://wsnl.stanford.edu/videos/envdisc/event_DiningTable1_T.avi

User Location and User Location and ActivityActivity


Human ActivityHuman ActivityHierarchical classification of activitiesHierarchical classification of activities

Location

Pose-based activities

Motion-based activities


Human ActivityHuman Activity1ta − ta 1ta + 2ta +

1ty − ty 1ty + 2ty +

Conditional random field (CRF) Classes: standing, sitting, lyingFeatures: height, aspect ratio

• Secondary activities:

• Standing pose + Global motion “Walking”g p g

• Sitting pose + Dining table + Local motion “Eating”

• Sitting pose + Living room + Gazing “Watching TV”


Human ActivityHuman ActivityMotion-based activities

Location Activity

Kitchen Cutting, scrambling, vacuuming, other

Motion-based activities

Dining table Eating, vacuuming, other

Living room Reading, vacuuming, other

S d T i di i hStudy room Typing, reading, vacuuming, other


Human ActivityHuman ActivityBag-of-features approachBag-of-features approach

Extract space-time interest points

descriptorscodebook, size = N

K-means clustering

episode,t seconds SVM

feature vector

activity classes

histogram feature vector for the episode


Human ActivityHuman ActivityMulti-camera fusion

Independent classification

Combined-view fusion (concatenate feature vectors)

Mixed-view fusion (mix feature vectors)Mixed-view fusion (mix feature vectors)

C. Wu, A. Khalili, and H. Aghajan, “Multiview Activity Recognition in Smart Homes with Spatiotemporal Features“, ICDSC 2010.

Use of space-time bag-of-features


Human ActivityHuman Activity

http://airlab.stanford.edu/videos/activity/airlab_activities_xvid.avi

Amir Khalili, Chen Wu, and H. Aghajan, “Hierarchical Preference Learning for Light Control from User Feedback“, CVPR 2010 Workshop on Human Communicative Behavior Analysis.


Behavior ModelingBehavior Modeling

Modeling transitions between location contexts

Dining

Kitchen

StudyRoom

gTable

LivingRoom

µ(Activiy{state}, Location),

δ(Living Room, Study Room)


Behavior ModelingBehavior Modeling

Storyline of the experimentsDining TableLiving Room Study RoomKitchen

Sample daily activity sequences

Dining TableLiving Room Study RoomKitchen

Watching Tv(9)

Studying(14)

Cutting(19)

Vacuuming(8)

Scrambling(14)

Cutting(15)

Scrambling(12)

Eating(32)

Studying(14)

Watching Tv

Scrambling(25)

Eating(9)

Watching Tv(8)

Studying(22)

Watching Tv(24)

Vacuuming(2)

Typing(6)

Studying(4)

Watching Tv(19)


Smart Homes: Prompting ServiceSmart Homes: Prompting Service

WellWell--Being ApplicationsBeing ApplicationsPRIME: Prompting Interactive Mobile Engagement systemTask and activity inference and user behavior modelingMobile device:

Smart Homes: Prompting ServiceSmart Homes: Prompting Service

Mobile device:Interactive prompting and instructions / tracking of responseCarrier of user behavior modelMaintain long term report on lifestyle and cognitive healthMaintain long-term report on lifestyle and cognitive health

Adaptive Smart HomesAdaptive Smart HomesAdaptive Smart HomesAdaptive Smart HomesLearning from User FeedbackLearning from User Feedback


Smart HomesSmart HomesUser comfort / ambient lighting and multimedia

Goal:• Minimize need for explicit user interaction / query• Minimize cost of service (energy)• Minimize cost of service (energy)

User-centric design:Accumulate knowledge about user lifestyle, preferences in service parameters


Occupancy-based context-awareSmart HomesSmart Homes

Occupancy-based, context-awareContextual data: Real-time activity, history / behavior model

Adaptation to user settings and preferences

Monitoring patient events Measuring exercise routinesMonitoring patient events Measuring exercise routines

Energy-efficient light control based on user location and activity


Occupancy reasoning for dynamic lighting control:

Smart HomesSmart HomesOccupancy reasoning for dynamic lighting control:

– Adjust light level based on location and activity of the user– Minimize energy use while keeping the intensity (utility function) at comfort level

Smart cameras for user sensing:– Simple in-camera bounding box extraction– Low-bandwidth communication Light intensity Low bandwidth communication g y

utility functions for different poses

http://wsnl.stanford.edu/videos/occupancy/ video_huang_intensity.avi


Smart Homes Smart Homes –– Ambience ControlAmbience Control

Making the system user-centric:Learn and adapt to user preferences over timeLearn and adapt to user preferences over time

User-centric design choices:F ll t t d t OR th h– Fully automated system OR query the user on every change

– Explicit input / feedback from the user OR implicit feedback

Learn user’s preference (user modeling)Learn user’s preference (user modeling)– Learn from explicit / implicit feedback– Explicit: Reward / penalty direct scoring optionp p y g p– Implicit: changing the offered service, leaving the area

Add other services: background music, ambient lighting


Adaptive Smart HomesAdaptive Smart HomesG lGoal:• Adaptation to user preferences• Activity-level granularity

User Location, Activity

Context Module

Activity level granularity• Minimize energy usage

ActivityOBSERVER

Method (user-centric design):U P fil

knowledge base

Adaptation Module Environment

User Feedback( g )

• Measure real-time user context• Learn user preferences via feedback• Develop behavior model for user

User ProfileMODEL

FeedbackINTERFACE

• Develop behavior model for user• Contextual data:

Real-time activity, location, time, user profileAmbience

ControlACTOR

Reinforcement Learning

LEARNER

Energy Cost


Adaptive Smart HomesAdaptive Smart Homes

LOCATION

User Location, Activity

Context Module

ACTIVITY

ActivityOBSERVER

Light intensity utility

User-adapted Utility Functions

U P fil

knowledge base

Adaptation Module Environment

User Feedbackg y y

functions for different contexts

User ProfileMODEL

FeedbackINTERFACE

AmbienceControl

ACTOR

Reinforcement Learning

LEARNER

Amir Khalili, Chen Wu, and H. Aghajan, “Hierarchical Preference Learning for Light Control from User Feedback“, CVPR 2010 Workshop on Human Communicative Behavior Analysis.

Energy Cost


Convergence of different services / heterogeneous sources (internet,

Smart Homes and Media ConvergenceSmart Homes and Media ConvergenceConvergence of different services / heterogeneous sources (internet, TV, gaming, social networks

Mobile device: content search delivery point for media access homeMobile device: content search, delivery point for media, access home sensor network (from inside home and outside), carry user profile

Ambience setting based on context and user preferencesAmbience setting based on context and user preferences


Smart Homes Smart Homes –– Ambience ControlAmbience ControlService Features ValuesService Features Values

AmbientGenre

Blues, Christian, Classical, Country, Dance, Electronic, Folk, Hip_Hop, Holiday, Jazz, Latin, New_Age, Oldies, Pop, R_n_B, Reggae, Rock

Music Mood Calm, Exciting, Happy, Neutral, Sad

Volume Silence, Low, Medium, High

P ttDigital Window

Pattern Cloudy, Starry, Fireworks, Solid

Color Purple, Blue, Gray, Green, Orange, Red, Yellow

Brightness Bright, Medium, Dark, Off

AmbientMusic (‘Genre value’, ’Mood value’, ’Volume value’)

DigitalWindow(‘Pattern value’, ’Color value’, ’Brightness value’)


Smart Homes Smart Homes –– Ambience ControlAmbience ControlProvide services to user based on context

Action according to preferenceMutual discovery (element of randomness)

– State = (time, location, activity)– Service: ambient music, digital window

Environment(Home and User)

Sensory

(element of randomness)

Sensory Motor ActorUser preference

Attribute = ([classical, nostalgic],[dim, starry])

Reinforcement Reinforcement LearningLearning

Ambient music Digital window

Q LearningState (time, loc, activity)

User feedback User Modelaction a(t)

s(t)

Ambient music Digital window

User feedbackLike? / dislike?

QoS – Reward/Penalty R(t+1) 1s

2s

1l 2l 3l nlKsf

actio

n

ates

M satis

(light setting)l

Sta


Adapt to preference changeSmart Homes Smart Homes –– Ambience ControlAmbience Control

Adapt to preference change– Online and continuous learning (preferences may change)

– Role of randomness:• Allow mutual discovery (exploration vs. exploitation)• Preserve novelty of service • Mimic a human-like behavior (reliable but not 100% predictable)Mimic a human like behavior (reliable but not 100% predictable)

A. Khalili, C. Wu, and H. Aghajan, “Autonomous Learning of User’s Preference of Music and Light Services in Smart Home Applications“, Behavior Monitoring and Interpretation Workshop at German AI Conf, Sept. 2009.

Meetings of The FutureMeetings of The FutureMeetings of The FutureMeetings of The FutureUser Experience, Analytics,User Experience, Analytics,P li d P f R tP li d P f R tPersonalized Performance ReportsPersonalized Performance Reports


Meetings of the FutureMeetings of the Future

User Experience Analytics

Speaker • Non-laser pointer• Highlighting and actuation• Gesture control of content

Learning-based score• Gaze distribution

B d l h d

Real-Time Gesture Speaker Report

Assistance • Gesture control of content• Voice keywords commands• Smart documents

• Body language, hands• Walking pattern• Repetitive moves, habits

Meeting• Best camera view selection• Faces of the sides of Q/A

• Faces and interactions• Social signals, body language

View Selection Meeting Report

Meeting Manager • Gesture-based view control

• Geometry, places, gazes• Shared content control

• Participation balance, flow• Models and roles• Personal report card


Speaker Assistance SystemSpeaker Assistance System


Speaker • Non-laser pointer• Highlighting and actuation• Gesture control of content

Learning-based score• Gaze distribution

B d l h d

Real-Time Gesture Speaker Report

Assistance • Gesture control of content• Voice keywords commands• Smart documents

• Body language, hands• Walking pattern• Repetitive moves, habits



User feedback

User behavior adaptation

User feedback

Real-time gesture

User behavior model User

preferencesSpeaker report

Smart presentations

UserUser--centric centric behaviorbehavior


(Face angle + pointing gesture) Intention interpretation Algorithm switchingSpeaker Assistance SystemSpeaker Assistance System

Adaptation:User System

Context( g p g g ) p g g

Calibration-free system: User System

http://wsnl.stanford.edu/videos/gesture/pdemo.avi

“Non-Laser Pointer”

C. Wu and H. Aghajan, “Context-aware Gesture Analysis for Speaker HCI“, Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI), co-located with ECAI, July 2008.



User feedback

User behavior adaptation

User feedback

Real-time gesture

User behavior model User

preferencesSpeaker report

Smart presentations

UserUser--centric centric behaviorbehavior



Measurement pool

ObservationWalking Screen gaze

Face orientation distribution

Use of hands

Standing pose

Walking pattern

Hand gestures Sky gaze

Voice level / tone

Other repetitive gestures / habits

Speaker Report• Gestures and body language• Desired / undesired habits• Comparison with historic data



Measurement pool

History Observation Smart Document

Face orientation distribution

Use of handsUser behavior model

• Emphasis points

Document context

Standing pose

Walking pattern

• History of scored (un)desired habits

p p• Transitions• Interactions with

audience

Voice level / tone

Other repetitive gestures / habits

Speaker Report• Gestures and body language• Desired / undesired habits• Comparison with historic data


A t E l

Speaker Assistance SystemSpeaker Assistance SystemAspect Examples

ContentTopic; Logical organization;

Presentation slides (text picture animation)Presentation slides (text, picture, animation)

SpeechEmphasis; Repetition; Confidence;

Emotion; Pitch variation;

VisualGesture (use of hands); Posture; Movement;

Eye contact / gaze; Facial expression

Focusing on the visual cues:Focusing on the visual cues:• Head trajectory encodes global movement

• Face/head orientation encodes eye contactFace/head orientation encodes eye contact

• Near-body movement encodes use of hand


Use of face orientation to evaluate:

Speaker Assistance SystemSpeaker Assistance SystemUse of face orientation to evaluate:– Distribution of gaze towards the audience– Frequency and duration of looking at display– Sky gazing occurrences– Looking at the floor

WSNL - StanfordSemantic labeling of clusters



• User-centric design:• User as definer of a value metric:• User as definer of a value metric:

– Intuitive, high-level input by user

– Self evaluation system: • Allow subjective metric (user definitions of desired / undesired)

• Personal system (privacy)

• Training of a scoring system based on user’s own metric


Speaker Assistance SystemSpeaker Assistance SystemUser-centric learning of performance scores:– Training set is created from user’s lectures– User scores own performance in segmented clips (episodes)

• Intuitive, high-level scoring scheme

– The scoring function is trained based on user’s scores (e.g. 1 – 10)

Interactive Machine Learning

Scored Episodes

User-specific scoring Processing Unit

Vision Processing

Sensing / recording

EpisodesLearning Scoring

Function


Vision processing:Speaker Assistance SystemSpeaker Assistance System

p g– Face orientation classification

• A head detector localizes face• Speaker’s gaze angle is classified into 6 angles

0.2

0.4- Head orientation histogram

– Global movement:• Based on KF head tracker

150

200

250

300

x100

110

120

130

140

150

y

01 2 3 4 5 6

ased o ead t ac e– Amount of movement– Span of movement

– Hand movement:0 100 200 300 400 500 600 700 800 900

0

50

100

150

time

x

0 50 100 150 200 250 30050

60

70

80

90

100

x

y

• Local motion inside the ‘body box’– Frequency of motion near the body– Span of motion near the body– Frequency of motion far from the body

An 11-D feature vector is produced and used in score trainingT. Gao, C. Wu, H. Aghajan, “User-centric Speaker Report: Ranking-based Effectiveness Evaluation and Feedback”, ICCV 2009 THEMIS Workshop.



Meetings of the FutureMeetings of the FutureUser Experience Analytics

• Best camera view selectionF f th id f Q/A

• Faces and interactionsSocial signals body language

View Selection Meeting Report

Meeting Manager

• Faces of the sides of Q/A • Gesture-based view control• Geometry, places, gazes• Shared content control

• Social signals, body language• Participation balance, flow• Models and roles• Personal report card


Meetings of the FutureMeetings of the Future

Improve experience of remote participantsremote participants


Which view to stream to remote participants?

Meetings of the FutureMeetings of the FutureWhich view to stream to remote participants?

Speaker’s gesture, body languageInteraction with the audienceAd t t f f th t ti i tAdapt to preferences of the remote participant


Signals Models Roles

Meetings of the FutureMeetings of the FutureSignals

• Gaze direction• Eye contact

Models

• Interaction• Eye contact

Roles

• Driver / follower• Info seeker / giver

Personal Features Interpersonal Features Inferred Features

• Eye contact• Nodding• Posture• Use of hands• Body language

Eye contact• Dialogue• Mimicry• Influence• Consistency

D i

Info seeker / giver• Supporter / neutral /

challenger

• Walking • Dominance

• Statistical, anonymized

Team Report• Confidential

Personal Report

• Only pre-selected data types• User-selected features

Meeting Report• Synergetic / polarized / boring / balanced in interaction• Uniform participation / apathy• Adequate discussion / group-thinking

g p

Inference via User InteractionsInference via User InteractionsInference via User InteractionsInference via User InteractionsUser Activity as Source of ContextUser Activity as Source of Context


Case Study: Object RecognitionCase Study: Object Recognition

Environment and object discovery in smart homes based on user interactionsbased on user interactions

Interfacing computer vision with high-level inferenceContext-driven data and decision fusion

Camera priors multi camera fusionCamera priors, multi-camera fusionUser location and activityCommonsense-based logic

Framework for probabilistic reasoning data fusion feedback toFramework for probabilistic reasoning, data fusion, feedback to vision, user-centric learning methods


Object recognition remains a challenging vision problemEnvironment DiscoveryEnvironment Discovery

Object recognition remains a challenging vision problem– Manual:

• Labeling of objects in view at camera deploymentAppearance based / model based:– Appearance-based / model-based:

• Large training set, may fail with variations, update with new object designs• Many objects may not be relevant to user’s lifestyle and routines

Th l bj t d fi th bj t’ ti l• The way a user employs an object defines the object’s semantic role

A user-centric approach:• User activity as context to discover the environment• User activity as context to discover the environment• User gives semantic definition to each object the way s/he uses it• User behavior model as a by-product


Simple cases:

Direct (Simple) InferencesDirect (Simple) InferencesSimple cases: – Based on user interactivity over time, discover:

Where the floor isWhere the doors areWhere the cameras are w.r.t. each other

These can be established using short-term observationsHow to develop a structured reasoning system?How to develop a structured reasoning system?• Which interacts with, queries from, and guides the vision module• And which considers the challenges in vision processing


Structuring A Knowledge BaseStructuring A Knowledge BaseSemantic labeling of objects based on user interaction:

– Three logic types:

1. Direct logic Appear/disappear

SofaSitting + lyingUser

DoorUser

2 Sequential-actions logic Take object Put object EatingUser

Appear/disappear DoorUser

2. Sequential actions logicout in + wait Eating

Fridge Microwave

User

Kitchen context

3. Concurrent-actions logic Sitting & WatchingSofa

User

Kitchen context

TVLiving room context


Structuring A Knowledge BaseStructuring A Knowledge BaseUser

Watching gestureDropping action

posepose

Sitting action

Sequential Object behaviorConcurrent

Object behavior

pose

Sofa

TV

Object behavior

Chair

Concurrent

Reclined pose

Coffee tableRules embed functional models for objects

Concurrent

Rules embed functional models for objectswatching: :TVlying: :sofasitting: chair, sofa

User activity used to give weight to rulesuser never reclined on sofa before

seen reclined is perceived as a new event

99 WSNL - Stanford

Spatial / temporal relationshipsobjects used concurrently - sofa: :TV: :coffee tableobjects used sequentially - fridge: :microwave

Behavior models of the userdrops on sofa to watch evening news on TV


Fusion and Inference FlowFusion and Inference FlowUser activities

Walking(t), Sitting(t), Gazing(t), HandMotion(t), …InKitchen(t), InLivingRoom(t), …

0.99 Walking(t) ^ Floor(t-1) -> Floor(t) Domain knowledge

( ), g ( ),

g( ) ( ) ( )0.5 Walking(t) ^ Chair(t-1) -> Floor(t)0.9 Sitting(t) ^ NA(t-1) -> Chair(t) v Sofa(t)0.9 Lying(t) ^ Chair(t-1) -> Sofa(t)0.1 Lying(t) ^ Floor(t-1) -> Sofa(t)

Domain knowledge in MLN

y g( ) ( ) ( )………

Grounding weight

Markov Random Field

Feature function

Pr(query)Number of true groundings of a formula


Fusion and Inference FlowFusion and Inference FlowMarkov Logic Network: Make the constraints soft by assigning weights to themg y g g g

0.99 Walking(t) ^ Floor(t-1) -> Floor(t) 0.5 Walking(t) ^ Chair(t-1) -> Floor(t)0 9 Sitting(t) ^ NA(t-1) -> Chair(t) v Sofa(t)0.9 Sitting(t) NA(t-1) -> Chair(t) v Sofa(t)0.9 Lying(t) ^ Chair(t-1) -> Sofa(t)0.1 Lying(t) ^ Floor(t-1) -> Sofa(t)

………

MLN a model to handle:• Imperfect visual output• Rules with weights

C. Wu and H. Aghajan, “Using Context with Statistical Relational Models – Object Recognition from Observing User Activity in Home Environment”, ICMI-MLMI Workshop on Use of Context in Vision Processing, Nov. 2009.


Markov Logic NetworkMarkov Logic NetworkRules (constraints): encode domain knowledge in first-order logic syntaxWeights Weights

(soft constraints)(soft constraints)

Walk FloorSit Chair or SofaLie Sofa

1.00.50 8

Examples of Direct logic

Example of Concurrent-actions logicLie Sofa

Stand still in kitchen (preparing food)

Gaze TV0.80.6

0.6

Example of Sequential-actions logic

then Enter living room then Sit with hand motion (eating) Dining Table

UncertaintyDomain knowledge

Observations (vision output)Grounding (probabilities dj t d ith b ti Observations (vision output)

Walk (t) with Probability 0.7adjusted with observations over time)


Fusion and Inference FlowFusion and Inference Flow

Semantic processing

Event processingp g

Individual

Central unit Vision

processing

CAM 1 CAM 2 CAM n

cameras


Local ProcessingLocal Processing• Simple vision processing employed in each view

• Possible flaws in detectionPossible flaws in detection

• Frame based probability of pose between views• Frame-based probability of pose between views

• Different cameras may see the user concurrently• Different cameras may see the user concurrently• Decision fusion based on declared pose


MappingMapping• Map observed poses to spatial grid (e.g. walking)

• Evolve probabilities for each grid point over time with observations

WSNL - Stanford


Build up probability map based on pose at location

MappingMappingu d up p obab ty ap based o pose at ocat o

Walk => FloorSit => Chair or Sofa

Direct relations:

Recline => Sofa

Gazing => TV

Spatial relations:

• Can exploit spatial relationship between grid points


Context in ProcessingContext in Processing• Features used in pose analysis:

• Bounding box (size, aspect ratio), calibrated height

Camera priors as context:• Range of acceptable foreground bounding boxes

Two SVM classifiers to learn range of too large and too small bounding boxesU dj t f f i t t d l b li f t i i t

• Performance of past pose classification reports- Only cameras 2 & 5 can see lying pose- Camera 2 never reports lying correctly

C 5 h 34% ll 63% i i

Use adjacent frames for semi-automated labeling of training set

- Camera 5 has 34% recall, 63% precision

Rows: ground truthColumns: estimates


For each grid point:

Living Room ExampleLiving Room ExampleFor each grid point:

Evolve probabilities over time with observations

Time


Living Room ExampleLiving Room Example

Time


(1) SequentialSequential--actionsactions logiclogicKitchen ExampleKitchen Example

Stand still in kitchen (preparing food) then Enter living room then Sit with hand motion (eating) Dining Table(2)Sit ith h d ti i li i ( ti )

qq gg

1

Sit with hand motion in living room (eating)then Enter kitchenthen Stand still in kitchen (putting away plates) Sink

1

2Living Room

Dining table

Fridge

0.39

0.36

0.38

6

5.40m

3

5

6

Workspace

Sink0.28

0.30 0.38 0.44

4

Kitchen0.380.38

4

C. Wu and H. Aghajan, “User-centric Environment Discovery with Camera Networks in Smart Homes”, IEEE Trans. on Systems, Man, and Cybernetics Part A, 2010.


E i t DiE i t DiFloor Chair S fDirect logic example

Living Room ExampleLiving Room ExampleEnvironment DiscoveryEnvironment DiscoveryFloor Chair SofaDirect logic example

More chair area discovered

Possible TV locations Some possible TV locations are ruled out as more activities are observed

Concurrent-actions logicexample


C t k

Next StepsNext StepsCurrent work:– Incorporate explicit user query and feedback

Walk FloorSit Chair or SofaLie SofaGaze TV

1.00.50.80 6

Stand still in kitchen (preparing food) then Enter living room th Sit ith h d ti ( ti ) Di i T bl

Gaze TV0.6

0.6

then Sit with hand motion (eating) Dining Table

• When to query the user to confirm or disambiguate a result?When to query the user to confirm or disambiguate a result?• How to score user input vs. accumulated knowledge?• How to use user input to adjust the weight of the rules?j g

Interactive Machine Learning


Open QuestionsOpen Questions

How to model semantics, ontology, knowledge representation– History of observations– User input– Use of the internet, search-based, community-basedUse of the internet, search based, community based

definitions


Iterative DiscoveryIterative Discovery

Object ObjectS

knowledge base

Activity Object

knowledge base

Spatial relationPose relation

Primary AuxiliaryPrimary Object

inference

AuxiliaryObject

inference

Object ActivityMotion relation

knowledge base

Activity inference module Object inference module


User Modeling and AdaptationUser Modeling and Adaptation

Learn the user’s habit

tablecoffeeeating tablediningeating

2

1

⇒⇒

ww

deskeating 3 ⇒w

Summary and OutlookSummary and OutlookSummary and OutlookSummary and OutlookFuture Research DirectionsFuture Research Directions


Vision: Challenges & OpportunitiesVision: Challenges & OpportunitiesGeneric features: Can become unavailable or un-interestingGeneric features: Can become unavailable or un interesting

Opportunistic features

Frame scope: Frames can have zero or misleading information valueFlexible fusion window

Calibration: User-dependent functionObservation-based self calibration methods

Point-to-point accuracy metric: May not be relevant to end applicationApplication-driven metric

E i t d fi itiEnvironment definition: Manual, need to repeat upon movementObservations of user interaction

Fixed vision function: To avoid overloading the processor in real-time app.g p ppAlg. switching, Task assignment (active vision)

Uniform value for data: Misleading data can bias resultsConfidence level based on content and historyConfidence level based on content and history

Enabled by two-way interfacing of vision with higher processing layers


In vision processing:

Vision: Challenges & OpportunitiesVision: Challenges & OpportunitiesIn vision processing:– Use context (incl. multimodal sensing) to guide vision– Algorithm switching based on contextg g– Associate confidence with data

Multi-camera vision:Multi-camera vision:– Handling redundancy, methods of data fusion, role selection– Privacy management (smart cameras)y g ( )

Address network deployment challenges:Online calibration / Calibration free methods– Online calibration / Calibration-free methods

– Automated setup and environment discovery

I i i f f i i i h i d lInteractive interface of vision with reasoning modules – Develop behavior models for user, environment, etc.


Models and InterfacesModels and Interfaces

communicationmodes

behaviormodels

high-levelreasoning

graphic visualization

user modeling environment discovery

camera network model

abnormality detection

best view selection

3D reconstruction

avatarmapping textual report


user preferences and availability

observation validation

context interface

vision

pose / activity face / gaze motion pattern 3D model

WSNL - Stanford

vision

multi-camera hardware & network


Models, Context, FeedbackModels, Context, Feedback

camera network model(geometry, topology, roles, priors, confidence levels)

environment discovery(objects, temporal/affine

usage relations)

user model(routines, preferences,

emotional state)

abnormal event detection(activities, routines, accident,

control zones, crowds)

behavior modelscontext

semantic-level context


i

context

reasoningobservation validation


c o n t e x t i n t e r f a c e signal-level

context

vision


context

system levelvision

hardware & network

system-levelcontext


Well-beingApplication DomainsApplication Domains

Well beingMonitor all relevant activities to user’s health and well-beingInput from heterogeneous sensorsSocial interaction: query the user confirm with an expert demographic comparisonsSocial interaction: query the user, confirm with an expert, demographic comparisons

Energy efficient smart environmentsContextual data: Real-time activity, history / behavior model, user preferences

Smart buildingsReal-time occupancy maps for emergency evacuation and rescue guidanceResource utilization: space lighting walkways energy activity patternsResource utilization: space, lighting, walkways, energy, activity patterns

Ambient multimediaConvergence of different content, interrupt prioritiesAmbience and atmosphere based on context and user preferencesSensor-enabled applications in social networks

Smart conference roomsGestures and interactions, smart documents, internet search, broadcast TV, streaming video, switching to best views, history and behavior modelsAnalytics: speaker performance card, roles and interactions in meetings


Design methodology for multiple applications on the same systemNew FrontiersNew Frontiers

g gy p pp y– Context-aware multi-purpose smart spacesFixed + mobile camera networks

Calibration free methods– Calibration-free methodsHow to model semantics (use of observations, history, user input, internet)– Ontology, knowledge representation, generalizationHuman-in-the-loop methodologies– Level & type of user interaction and query– “Logical” user vs. discovery based on “experimenter” userSystems with adaptive behavior– Exploit “system user” adaptation in designEvent modeling:Event modeling: – Adaptive granularity in space, time (when to declare an event, levels of description)User Behavior change:

A lt f ki ith b ti ( i i ) t– As a result of working with an observation (vision) system– As a result of working with an adaptive system– How to influence and measure it?


ResourcesResources


St d t & C ll b t

CreditsCreditsStudents & Collaborators:

Chen WuAmir KhaliliHuang LeeJingyu CuiTianshi GaoStephan HengstlerItai KatzItai KatzNan HuAli Maleki-TabarArezou Keshavarz

Marleen Morbee, Linda Tessens (Ghent U., Belgium), Tommi Maataa (Philips, TU Eindhoven, Netherlands)

Ronald Poppe (Univ. of Twente, Netherlands), Jacopo Staiano (Unit. Of Trento, Italy)Ralph Braspenning, Aki Harma, Joyca Lacroix, Aart van Halteren (Philips Research)


ContactContact

http://airlab stanford eduhttp://airlab.stanford.eduaghajan @ stanford.edu

from sensor networks to smart environments and social networks

Documents