Activity, Audio, Indoor/Outdoor classification using cell phones

Hong Lu, Xiao Zheng Emiliano Miluzzo, Nicholas Lane

CS 185 Final Project presentation

Outline

• Motivation

• Results

• Conclusion

• Future work

• Demo

Motivation

• In our lab we are working on a project where we use sensors to determine the sensing presence of people, i.e.:– Activity– Audio analysis– Location (Indoor/Outdoor)

• We are interested in using off-the-shelf devices like cell phones (with the camera, microphone, and accelerometer)

Requirements

• The activity and audio inference must be done on the phone (to improve scalability and limit the amount of bytes sent over the air)

• Downside: limited hardware (computation resources) and software (programmability) capabilities

Let’s start with sound

Experimental methodology

• Over a period of one week we collected 250 audio samples of conversations and 250 audio samples w/out conversation

• 8 (>) people involved in different conversation settings

• The phone used to collect the samples was placed at different places (inside/outside pocket) and at different distances from the audio source

• The samples w/out conversation are taken indoors (office environment), outdoor in a quite location, and outdoor on the sidewalk (to catch the sound of cars, etc.)

Observing the FFT

Conversation

No Conversation

Sound feature vector

• We focus on the 250 – 750 Hz portion of the spectrum

• We focus on the – Mean: because we see that the power differs

in the 2 cases in this portion of the spectrum– Std Dev: we also see that the power

oscillates more in the conversation case than in the no conversation case

Mean and Std dev

Mean and Std dev distribution

Total Error

ROC curves

Misclassification = 31% Misclassification = 28.4%

K-means Discriminant analysis

Now location…

ProblemDetecting when the cell phone is indoor or outdoors. Use the limited sensing capabilities of the phone.

Why try to do this?

1.Health Sensing Simple less accurate UV and pollution exposure with value in the potential for wide spread sampling

2.Context-dependant sensing queriesIndoor/outdoor classification is one useful criteria of describing conditions when (and when not) sense.

3.It seemed interesting to try

Intuition of an approach

• Sensor values will exhibit different characteristics depending on if the phone is indoors or outdoors.

• A key problem == Limited Sensors Available: – Different sensors would have made this simpler– Available: WiFi, GPS, Sound, Camera, GPRS etc.

• We tried feature vectors formed by using: – GPS + Radio Patterns (from WiFi)

• Various features tried that were based on:– WiFi AP Counting

• (assumption: lower density outdoors vs indoors)

– Sum of WiFi AP Neighborhood RSS • (assumption: lower signal strengths seen outdoors)

– GPS Satellite Signal Lock • (assumption: signal lock, which is effected by LOS issues, will occur

less often indoors than outside)

Hope from weak discriminators?

KEY: GREEN == OUTDOORS BLUE == INDOORS

Data Set: 1 user, 2,000 data points collected; 1 sample per minute. Sampling WiFi and GPS based measurements.

Groping for a modelTechnique Used: Logistic regression

Tested many variants of the inputs to find effective feature vectors.Windowed Variables – [RSS Sum, AP Count, GPS Flag]Variance of Windowed Variables – [RSS Sum, AP Count, GPS Flag]

Multiple variants of model inputs failed to provide sufficient discriminative power.

ANOVA test showed little reduction in deviance.We favored a simpler model over the complex one.

Checked that normal distribution existed in feature vectors

The Modelclassifier_label_param ~ ap_count + gps +

rss_sum

(Intercept) ap_count gps rss_sum -0.23107255 2.49738687 0.46060383 -0.03293664

Significance of the co-efficients? (Are they really different from zero?)

ap_count < 2e-16 ***gps 0.00358 ** rss_sum < 2e-16 ***

Kicking the tires of the model

Empirical performance

dubious hypothesis results not impressive but interesting focused on other parts.

Mini-conclusion

Outdoor Indoor

Outdoor 1636 94

Indoor 259 468

Accuracy85.6%

Now activity…

Data collection

• 3-axis accelerometer embedded in the Nokia Sport Phone (in the future many phones will have an accelerometer)

• Sampling rate = 37Hz

• The phone can be clipped on the belt or carried inside the pocket

• Data labeled by the user

The raw data

0 2000 4000 6000 8000 10000 12000

-300

-200

-100

0

100

200

Time

Standing

0 2000 4000 6000 8000 10000 12000 14000 16000

-2000

-1000

0

1000

Time

Walking

0 1000 2000 3000 4000 5000 6000-4000

-2000

0

2000

Time

Running

X

YZ

X-axis – BLUE

Y-axis – GREEN

Z-axis – RED

From the 3 axis of the accelerometer

Feature Extraction

• Simply the mean and standard deviation

of each axis

• Window size = 128 data points

• Lightweight for running on the phone

PCA

• Use centerized PCA to get principal component projection matrix:

• Each column contains the coefficients for one principal component. The columns are sorted by decreasing component variance.

-0.0150 -0.0862 0.0762 -0.0316 -0.0801 0.9895 -0.0086 0.3733 0.4937 -0.7741 0.1314 -0.0197 0.0898 0.6502 -0.7286 -0.1438 0.0665 0.1149 -0.8425 -0.2966 -0.3070 -0.3274 0.0082 -0.0248 -0.4889 0.5694 0.3288 0.4243 -0.3854 -0.0008 -0.2067 0.1352 0.1310 0.3031 0.9074 0.0817

The 2 dimensional example

• Use the first 2 components, leave out the others.

-600 -500 -400 -300 -200 -100 020

30

40

50

60

70

80

standing

walkingrunning

Conclusion

• We wanted to do more than we could do

• Collecting the data took long time

• Programming the phones can be challenging

• The phone’s hardware and software platform limit the applicability of off-the-shelf data mining techniques

Now the demo!!!

Future work

• Increase the number of activities we can detect (including sitting, cycling, climbing stairs, etc)

• Augment the audio classifier to be able to do speaker recognition, party detection, type of noise (car, wind, etc)

• Use the cell phone’s camera and light detector to make the outdoor/indoor classifier more robust

Thank you!