location and context awareness

11© 2010-2011-2012 Daniel P. Siewiorek

Mobile Computing

Location and Context Location and Context AwarenessAwareness

Dan SiewiorekDan Siewiorek

June 2012June 2012


Mobile Computing

Outline

Distraction Context Aware Computing Location Activity Recognition Applications Research Challenges


Mobile Computing

Outline



Mobile Computing

710

1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996

310

410

510

610

Year Samples Introduced

4004 8080

6800

8086

80286

80386

80860 80486

PENT.

6802

6801

68020

68030SLOPE = 10X INCREASE

IN 7 YEARS

68040

PENT. PRO

Num

ber

of

devi

ces

(Source: Walt Davis, Motorola)

Transistors per Processor

Moore’s Law Reigns Supreme


Mobile Computing

Disk Capacity



Mobile Computing

Cost per Megabyte



Mobile Computing

Glaring Exception

Adam & Eve 2000 AD

Human Attention


Mobile Computing

Distraction Matrix


Mobile Computing

Outline



Mobile Computing

Context Aware Computing

Applications that use context to provide task-relevant information and/or services

Context is any information that can be used to characterize the situation of an entity (person, place, or physical or computational object)

Contextual sensing, adaptation, resource discovery, and augmentation


Mobile Computing

Context Aware Service Examples

Primary Services» Location

» Biological measures (heart rate, breathing)

» Position of limbs

Derived Services» Difficulty in performing activity

» Amount of activity for elderly

» “Is Bob coming to the meeting”» Match Making (location, activity, skill level)


Mobile Computing

Outline



Mobile Computing

Location Sensing Parameters

Physical (x,y,z) versus Symbolic (room) Absolute (shared reference grid) vs Relative Localized Location Computation - where calc Object Recognition Accuracy, Precision

» Distance, Distribution Scale

» Number of objects per unit infrastructure per time interval

Cost


Mobile Computing

Fluctuating signals for a single-location over time in the Wean

Linux Cluster


Mobile Computing

Combined data-points from Linux and Windows Cluster in Wean Hall


Mobile Computing

Location Sensing Approaches

Triangulation» Lateration – multiple distance measurements

between known points» Angulation – angle or bearing relative to points

with know separation Proximity

» Nearness to known set of points Scene Analysis

» Identify relationship to know points


Mobile Computing

Triangulation Technologies

Technology Resolution Example

InfraRed (IR) Room Active Badges

UltraSound cm Bats, Cricket

Visual Light cm Easy Living

Wireless LAN m RADAR, Wireless Andrew


Mobile Computing

Location Service Architecture Comparison

Four main types of location architectures (Centralized Push, Centralized Pull, Distributed Push, Distributed Pull)

Location aware messaging as a representative application

Quantified data flow requirements and messages for location based application

Centralized pull model performs better than distributed for location aware messaging


Mobile Computing

Location Service Architecture Alternatives


Mobile Computing

Location Service Architecture Data

Rates

Location to Server Server to App Client to Client Total Data Rate

Centralized-Pull 0.17 Mbps 2.56 Mbps N/A 2.73 Mbps

Centralized-Push (Time) 0.78 Mbps 2.56 Mbps N/A 3.34 Mbps

Centralized-Push (Event) 0.01 Mbps 2.56 Mbps N/A 2.57 Mbps

Distributed-Pull N/A N/A 93.80 Mbps 93.80 Mbps

Distributed-Push (Time) N/A N/A 14.81 Mbps 14.81Mbps

Distributed-Push (Event) N/A N/A 3.47 Mbps 3.47 Mbps

Comparison of Architectures using Instant Messaging Application Polling/Push Frequency = 1 min ~ 3,000 wireless clients

48 Buddies per User


Mobile Computing

Systems Issues (Bats) Aesthetics Distribution of sensors by space usage Physical/symbolic boundaries

» Overlap False negatives

» Not wearing Quiet Zone

» Not tracked Cycle: user participation decreases

application degrades reduced incentive to participate


Mobile Computing

Outline



Mobile Computing

Activity Recognition Through Machine

Learning

.


Mobile Computing

Model Generation Variables

Window Sizes» 4, 6 Seconds

Features Extracted Model

» k nearest neighbors

» clustering

» Support Vector Machine (SVM)

‘Leave-one-out’ cross validation for optimization and testing

Features Value

mean 10.04361

stdev 0.05040

variance 0.00254

rms 10.04373

cumulative sum 954.14258

mean absolute deviation 0.04104

mean crossing rate 64


Mobile Computing

Feature Space After Linear Discriminant Analysis (LDA)

Transformation


Mobile Computing

Feature Subsets and Classification for Body

Positions

# Features CPU Cycles Time in μs Classification Accuracy for Body Position

wrist pocket bag necklace shirt belt

F1 All features, all sensors 56242 953.6 87.1% 85.2% 92.8% 86.8% 89.5% 87.0%

F2 All features from accelerometer X 14731 249.8 68.4% 78.6% 85.2% 74.3% 76.1% 72.6%

F3 All features from accelerometer Y 14731 249.8 83.2% 75.4% 86.4% 61.3% 70.1% 70.8%

F4 All features from light 14731 249.8 55.0% 16.7% 18.0% 61.7% 52.2% 48.8%

F5 All features from accelerometer XY (x2 + y2 ) 15049 255.2 76.6% 79.5% 87.2% 72.6% 78.0% 77.2%

F6 prcy(3), rmsxy, prcy (20), prcy (97), rmslight, madx, meany, prcy (10)

12114 205.4 87.0% 80.1% 86.5% 78.6% 79.6% 84.2%

F8 prcy (3), iqry, prcy (10), prcy (97), madx 7651 129.7 82.0% 62.4% 68.9% 56.6% 69.8% 71.7%

F9 rmsxy, qrtx, rmsx, madxy, meanxy 10746 182.2 77.3% 78.2% 80.9% 72.3% 75.4% 76.5%


Mobile Computing

Activity Recognition Accuracy at Body

Locations


Mobile Computing

Accuracy Classification Recorded Over 100

Minutes


Mobile Computing

Audio and Light Sensor Clustering


Mobile Computing

New Sensor Types Increase Range of Activity Recognition

Physi

olo

gic

al Senso

rs

Accu

racy


Mobile Computing

Cell Phone Activity Recognition Deployment


Mobile Computing

Data Quality

In a real-time wireless environment data centric systems are plagued with lost packets and corrupt data.

A dynamic sensor architecture can mitigate these problems.


Mobile Computing

Handling Multiple Sensors

Centralized Architecture

Low Bandwidth Architecture

Dynamic Architecture

Raw Sensor Data

Feature Extraction

Classifier

Decision

Master Device

Sensor Device

Master Device

Sensor Device

Master Device

Sensor Device

Aggregation of N sensors takes place between the two devices.


Mobile Computing

Centralized Architecture

Classifier is static.» Requires knowledge of

sensors devices available (N).

Bandwidth utilization will be higher (~500 B/s per sensor) sending all raw data.

Raw Data Aggregator must deal with memory intensive data sets (~1-10 MB)

Raw Sensor DataIndicates wireless link

~500 B/s

Feature Extraction

Classifier

Decision

Raw Data Aggregator

Sensor Device 1

Master Device

~1-10 MB/window

~1 KB/window

~4 B/window

Raw Sensor DataSensor Device N

…

~500 B/s


Mobile Computing

Coping with Data loss in the Centralized

Architecture Process 1:

» Buffer data on master device with the Raw Data aggregator.

» Forward incomplete windows at least 2/3 full to Feature Extraction.

» Ex: Forward a set of 40 data points when 60 are expected.

Indicates wireless link…

Raw Sensor Data

Feature Extraction

Classifier

Decision

Raw Data Aggregator

Sensor Device 1

Master Device

~500 B/s

~1-10 MB/window

~ 1KB/window

~4 B/window

Raw Sensor Data

Sensor Device N

~500 B/s


Mobile Computing

GSR Temp ECG Respiration0

100

200

300

400

500

600

700

800

900

Data Windows Collected per Sensor Type

Subject08Subject14Subject16Subject20

Sensor Type

Num

be

r o

f Da

ta W

ind

ow

s C

olle

cte

d

Coping with Data loss in the Traditional Architecture

Ex: With subject 20 a maximum of 604 windows could be classified on.

Process 2: Classify only on windows in which all sensors have an output.


Mobile Computing

Low Bandwidth Architecture

Lower wireless bandwidth compared to centralized architecture

Indicates wireless link

~200 B/window

Sensor Device N

Raw Sensor Data

Feature Extraction

Classifier

Decision

FeatureAggregator

Sensor Device 1

Master Device

~1KB/window

~4 B/window

…Raw Sensor Data

Feature Extraction

~200 B/window

~500 B/sec~500 B/sec


Mobile Computing

Dynamic Architecture Fuse locally

made classifications from multiple sensors.

» N is dynamic.

Confidence information, the probability of each context, is transmitted to Fuser. (~80 B/window)

Sensor Device N

Raw Sensor Data

Decision

SensorFuser

Sensor Device 1

Master Device

~4 B/window

…

Feature Extraction

Classifier

Raw Sensor Data

Feature Extraction

Classifier

~80 B/window~80 B/window

~200 B/window ~200 B/window

~500 B/sec ~500 B/sec

Indicates wireless link


Mobile Computing

Comparison to Centralized Architecture

Utilizes dynamic sensor set» Increased Accuracy

» No static classifier in a central location.

Utilizes heterogeneous algorithms» Best techniques can be used on a per device

basis to address:– Power constraints

– Computation constraints


Mobile Computing

ErgoBuddy – Experimental Setup

11 Subjects» Approximately 22 hours of

data total.

7 Sensor Body Locations» Ankle, Arm, Back, Handheld,

Holster, Lanyard, Wrist

10 Activities» Sitting, Standing, Lifting,

Walking, Running, Carrying, Sweeping/Mopping, Stairs, Laddering, Carting


Mobile Computing

Experimental Issues

7 Wearable Sensors for activity recognition communicating over Bluetooth.

Approximately 10% packet loss per sensor with current implementation.

» Low Bandwidth Architecture Reliability ~48%– 1 packet lost = missed classification

– Probability one of seven sensors is down @ 90% reliability:

– 0.9 ^ 7 = 0.48

» Dynamic Architecture Reliability ~99.99%– 1 packet lost = continue with N-1 other sensors.

– Probability all seven sensors are down @ 90% reliability:

– 1-(.1 ^ 7) = .999999


Mobile Computing

Model Generation 2 Final Window Sizes

» 4, 6 Seconds

7 Models» 1 model for each body

location

5 Fusion techniques 0%-90% simulated packet

loss environments ‘Leave-one-out’ cross

validation for optimization and testing

Features Value

mean 10.04361

stdev 0.05040

variance 0.00254

rms 10.04373

cumulative sum 954.14258

mean absolute deviation 0.04104

mean crossing rate 64


Mobile Computing

Fuser Technique – Lossless Environment

Fuser Method Accuracy

Average of Probability 88.92%

Product of Probability 59.09%

Straight Vote 83.66%

Max Probability 79.34%

Rank 87.89%Raw Sensor Data

Decision

SensorFuser

Sensor Device 1

Master Device

…

Feature Extraction

Classifier

Raw Sensor Data

Sensor Device N

Feature Extraction

Classifier


Mobile Computing

Resiliance of Static vs Dynamic Classifiers in Lossy Environments

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Packet Loss Rate

Cla

ssifi

catio

n A

ccu

racy 7 Sensors Static

7 Sensors Dynamic

6 Sensors Dynamic

5 Sensors Dynamic

4 Sensors Dynamic

3 Sensors Dynamic

2 Sensors Dynamic

1 Sensor Dynamic

Performance Results

With same number of sensors in a lossless environment fusion yields results 2% worse than a model with access to all sensor’s raw data.


Mobile Computing

Conclusions

In all lossy environments, 10%+, there was better performance using fusion.

» 35% accuracy increase in an environment with 50% packet loss.

The number of sensors can be reduced in low loss environments for power and bandwidth savings.

» For our experiment 3 sensors was ideal.

This technique can also be applied to systems of heterogeneous sensors.


Mobile Computing

Outline



Mobile Computing

Context Sensing

Basic context» Location

» Orientation

» Audio samples from the user’s environment

» Static data

» History of user context

Multiple sensors can be used to infer user’s intent» Wireless Network Card, Digital Compass,

Thermometer, Camera


Mobile Computing

Example Applications

Notification » Alert a user if they are passing within a certain distance of a task on

their to do list.» SenSay Context Aware Cell Phone

Meeting Reminder » Alerts a user if they are in danger of missing a meeting.

Activity Recommendation » Recommends possible activities/meetings that a user might like to

attend based on their interests.

Proactive Assistant» Answering questions about user’s intent» Proactively preparing user’s workspace based on usage patterns and

behavior

Matchmaking» Locating an entity based upon expertise, skills, proximity and/or

availability


Mobile Computing

Distraction Matrix for Portable Help Desk


Mobile Computing

Technology

Location Service» Use multiple sources to calculate location (e.g.

wireless access point triangulation, ceiling photo match)

» Give applications simple form

Transcoders» Translate data to form useful on device

Manage Network Disconnects Persistent Proxies for Devices, Users

» Allow policies to be set

» Remove burden from individual applications


Mobile Computing

Handy Andy Architecture

InfrastructureLogin/Logout

Waldo PhD Idealink Stalker

User Proxy

Device Proxy

Itsy Jornada Other Devices

SpeechEncode/Decode

Database

Service

Infrastructure

Device

Device Proxy Device Proxy


Mobile Computing

PhD Features and Interaction

User’s List:• Items can be added,moved, and removed• Only “checked” items appear on the map

Description:• Information on the currently selected item • Dynamic information automatically updated

Map:• Dynamic information automatically updated

Map Controls: Zoom & Pan


Mobile Computing

PhD (Personal help Desk)


Mobile Computing

PhD Control of Preferences


Mobile Computing

Virtual/Real Synchronously Sharing Information

Real Sites Virtual Sites Example

One One Training

Many One Virtual Control Room

One Many Collaboration

Many Many Command and Control


Mobile Computing

Virtuspherehttp://www.virtusphere.com

10-foot hollow sphere that rotates freely in any direction according to the user’s steps.

Wireless, head-mounted display allows user to walk and run being immersed into virtual environment.

User movement replicated in the virtual environment.


Mobile Computing

System Help

CMU SCS Computing Facilities DB Matchmaking : Expert to Problem Facilities people have certain expertise Users report problems Performs Matchmaking Assigns expert to the problem Gets reply/confirmation from expert


Mobile Computing

MatchMaking (MM)

Obtain a list of most relevant experts for this problem

Find out which of these experts are available and if available, after how much time (mean and variance)

Find time to reach location of the problem Using Rules, choose the best expert

» time (mean, var), expert busy?, expert’s score


Mobile Computing

Contexts Used

Location of the expert Location of the problem Expert profiles -skills Expert availability, current involvement (busy) Time spent on the problem so far History of maintenance (problems experts) Other simultaneous problems Time of the day


Mobile Computing

0.01

0.1

1

10

100

0 250 500 750 1000 1250 1500 1750 2000

Number of Client Apps.

Wa

it T

ime

in Q

ue

ue

(s

ec

.)

0

20

40

60

80

100

120

140

160

180

12:00 AM 4:00 AM 8:00 AM 12:00 PM 4:00 PM 8:00 PM 12:00 AM

Time of Day

Infe

cted

Wir

eles

s U

sers

U1

U2

U3

U4

G1

G2

F1

F2

S1

S2


Mobile Computing

Major Components


Mobile Computing

Locator@CMU

Implemented on Wireless Andrew» ~1,000 APs» ~5,000 peak concurrent users

Centralized-Pull architecture using relational database

Provides omniscient view of network usage


Mobile Computing

Mobility on Wireless Network

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

1 2 3 4 5 6 7 8 9 10+

Mobility (Access Points per Day)

Per

cen

t of

Use

rs

2003 2005 3.012.012.812005

2.752.182.852003

Standard Deviation

MedianMobility

AverageMobility

3.012.012.812005

2.752.182.852003

Standard Deviation

MedianMobility

AverageMobility


Mobile Computing

Location-Based Applications

Where are users now and where have they been (past/present)» Contact/Spread Tracking

Where were users (past) » Unknowing Bystander Service

Where will users be (future)» Crowd Predictor


Mobile Computing

Applications Contact/Spread Tracking

Scenario: Someone is ‘infected’ how many people do they spread the disease too in various situations?

Divided wireless users into infecting agents and general users.

Selected 10 infecting agents (4 undergrads, 2 grads, 2 faculty, and 2 staff)


Mobile Computing

Contact/Spread Tracking

Direct-Primary

0

20

40

60

80

100

120

140

160

180

12:00 AM 4:00 AM 8:00 AM 12:00 PM 4:00 PM 8:00 PM 12:00 AM

Time of Day

Infe

cted

Wir

eles

s U

sers

U1

U2

U3

U4

G1

G2

F1

F2

S1

S2


Mobile Computing

Applications Unknowing Bystander

Scenario….. An event happens at location X. People nearby might not even be aware but can have valuable information.

Event Examples:» Crime (Burglary/Theft/Murder, etc.)» Lost item, pet, or person

Possible Users» Police, Homeland Security (Citizen Watch Corps)» Individuals

Used campus crime data to determine how many network users near area and could be potential witnesses

What percent of time would there be a potential witnesses?


Mobile Computing

Unknowing BystanderResults

15 of the 16 crimes had potential witnesses Average value of 12.8 for potential witnesses Median value of 4.5 for potential witnesses Chance of at least 1 witness for 4.5 witnesses with

likelihood of 5% (21%); 10% (38%); 33% (83%)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Number of Wireless Clients

Pro

b. o

f a

t L

ea

st

1 W

itn

es

s

0%

1%

2.50%5%

10%

15%

25%

33%

40%50%

Likelihood of Wireless User Being a Witness


Mobile Computing

ApplicationsCrowd Predictor

Use information from historical data to populate an application to predict future crowds at a location (Neural Network)

Can be used by organizations to find best spot to setup table

Allow for other limited criteria (such as type of space, time of day, day of week)


Mobile Computing

Applications Crowd Predictor - Metrics

Select 12 different Access Points » 3 in Classroom Areas» 3 in Office Areas» 3 in Public Areas» 3 in Dorm Areas

Predict wireless crowds at 12 test Access Points at 5 different day/time combinations and compare to observed results

Look at effect of time of day, day, and type of area predicting


Mobile Computing

Applications Crowd Predictor

Mudge 211

0.00

2.00

4.00

6.00

8.00

10.00

12.00

11/8/200511:00AM

11/9/20052:20PM

11/10/200510:40PM

11/11/20054:30PM

11/12/20051:45PM

Wir

eles

s U

sers

Predicted

Actual 53.3%

46.7%

59.7%

40.3%

58.6%

41.4%

62.6%

37.4%

0%10%20%30%40%50%60%70%80%90%

100%

Inp

ut

Fa

cto

r Im

po

rta

nc

e

Office Public Classroom Dorm

Area Type

Time of Day Importance Day of Week Importance


Mobile Computing

Outline



Mobile Computing

Context Aware Computing Research Challenges

When and How to Interrupt Privacy of Data

» What information is collected

» Who can access information

» How long information is stored

How User Specifies Preferences on Data Availability

User Attention» Charge Market Value to those demanding attention

» Combine theories from social science, cognitive science, and economics

location and context awareness

Documents