where dsp meets ubicomp

Where signal processing meets ubiquitous computing:

the role of DSP in the vision of UbiComp; Sustainability, Health, and

User Interfaces

Eric Larson

Graduated Oklahoma State

University

Signals, Systems, Controls

2006

2011

2007

2008

2009

2010

MS Thesis: Subjective Image Quality Prediction

1st Paper: on image texture analysis3rd Paper: on predicting

image quality

5th Paper: on evolutionary computation / facial

analysis

(shift in interests…)

Started at University of Washington

time series analysis…

User Interface and Touch Sensing

HydroSense

Mobile Health Sensing

Religion:Language:Hobbies:

mathematics

MATLAB, C++

-filtering

-statistics

-doing FFTs by hand

-finding applications…

the DSP student

Religion:Language:Hobbies:

passion for UbiComp

everything but MATLAB

-writing code in five

languages

-soldering

-machine learning

-user studies

-picking an app…

the UbiComp student

toaster?

water fixture

classification

TED for water

health sensing

from phone

user interface sensing

TED for water

water fixture

classification

health sensing

from phone

user interface sensing

water scarcity

water

• single, screw-on sensor

• senses pressure in real time

hydrosense

Froehlich et al., UbiComp2009; Larson et al., PMC2010

brief plumbing tutorial

water towerwater tower

plumbing primer

water tower

plumbing primer

water tower

incoming cold water from supply line

water tower

pressure regulator

water tower

utility water meter

pressure regulator

incoming cold water from supply line

water tower

pressure regulator

water tower

utility water meter

pressure regulator

incoming cold water from supply line

water tower

plumbing layout

water tower

incoming cold water from supply line

utility water meter

pressure regulator

water tower

closed pressure system

water tower

thermal expansion

tank

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

hot water heater bathroom

2

dishwasher

incoming cold water from supply line

kitchen

water tower

closed pressure system

water tower

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

bathroom 2

kitchen

dishwasher

incoming cold water from supply line

hot water heater

thermal expansion

tank

hot water heater

water towerwater tower

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

hot water heater bathroom

2

dishwasher

incoming cold water from supply line

toilet flushed

toilet

kitchen

thermal expansion

tank

water towerwater tower

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

hot water heater bathroom

2

kitchen

dishwasher

incoming cold water from supply line

toilet

kitchen sink cold

open

kitchen sink cold

thermal expansion

tank

water towerwater tower

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

hot water heater bathroom

2

kitchen

dishwasher

incoming cold water from supply line

toilet

kitchen sink hot

open

kitchen sink cold kitchen

sink hot

thermal expansion

tank

water towerwater tower

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

hot water heater bathroom

2

kitchen

dishwasher

incoming cold water from supply line

kitchen sink hot

open

thermal expansion

tank

kitchen sink hot

water towerwater tower

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

hot water heater bathroom

2

kitchen

dishwasher

incoming cold water from supply line

40

50

60

70

80

Cold Line Pressure(Hose Spigot)

0 94.5

time (s)

psi

open

close

bathroom sink pressure signal

40

50

60

70

80

Cold Line Pressure(Hose Spigot)

0 94.5

time (s)

psi

open

close

bathroom sink pressure signal

flow volume

Poiseuille’s Equationpressure drop of laminar water flow in a pipe:

Q = Volumetric Flow RateΔP = Pressure Dropr = Pipe Radiusl = Pipe Lengthµ = Fluid Viscosity

Blasius’ Equation

pressure drop of turbulent water flow in a pipe:

Q = Volumetric Flow RateΔP = Pressure Dropr = Pipe RadiusL = Pipe LengthRe = Reynolds Numberρ = Fluid Density

Bucket Trials

• 9 Homes• 41

Fixtures• 541 trials

Flow Rate (gpm)

Pres

sure

(psi

)

Stabilized Pressure vs Flow Rate Residence 6

Stabilized Pressure vs Flow Rate Residence 7Stabilized Pressure vs Flow Rate Residence 7

Stabilized Pressure vs Flow Rate Residence 7

44

46

48

bandpass

derivative

(psi/s)time (s)

smoothed

pressure(psi)

0 5 10 15 20 25 30 35 40 4542

44

46

48

50

raw pressure

(psi)

-4

0

4

a new featuremax resonance

max amplitud

e

Max Amplitude vs Flow Rate Residence 7

Kitchen SinkBasement Sink

Bathroom Sink

Basement

Shower

Upstairs Shower

Upstairs Bath

stable pressure (psi)

max a

mp

litud

e

e-kNN

*Garcia, Gupta, and Chen (2008)

convex hull

linear regression

feature 1

featu

re 2

test point

pressure

reso

nance

e-kNN results

median error: within 0.15 gpm

80% within 0.5 gpm

gpm% o

f act

ual

volu

me

Neptune T-10 Water meter

error for e-kNN (10-fold cross val.)error E-kNN only kitchen + bath

volume prediction

hot vs cold

• max resonance only• ICA• median error: 0.2 gpm• 75th percentile: 0.42

gpm

TED for water

water fixture

classification

health sensing

from phone

user interface sensing

toilet78.4

gallons

shower62.4

gallons

bath6.5

gallons

today‘s usage

bathroom sink 1

4.2 gallons

bathroom sink 2

0.8 gallons

toilet78.4

gallons

shower10.4

gallons

bath0.0

gallons

bathroom sink 1

1.2 gallons

bathroom sink 2

0.8 gallons

shower52.4

gallons

bath6.5

gallons

bathroom sink 1

3.2 gallons

bathroom sink 2

2.4 gallons

today‘s usage: hot vs cold

hydrosenseexample pressure waves

bath open

dishwasher open

upstairs toilet flush

downstairs toilet flush

downstairs shower open

kitchen sink cold open

kitchen sink hot open

toilet

natural water use

70

50

30pre

ssu

re (

psi

)

kitchen sink kitchen sinkbathroom

sink

water tower

thermal expansion

tank

hosespigot

utility water meter

pressure regulator

laundry

bathroom 1

hot water heater bathroom

2

dishwasher

incoming cold water from supply line

kitchen

data collection

totals

days 33 33 30 27 33 156

events 2374 3075 4754 2499 2578 14,960

events/day 71.9 93.2 158.5 92.6 78.1 95.9

compound 22.2% 21.8% 16.6% 32% 21.3% 21.9%

5-week dataset

signal behavior

algorithm: bayesian inference

bath

toilet

unclassified event

term(i) signal features

shower

event library

unclassified event

term(i) signal features

shower

toilet

bath

toilet

pressure drop=flow rate

term(i) signal features

shower

toilet

bath

toilet

resonance tracking

unclassified event

sig

nal

tran

sform

sunclassified event event library

toile

t

unclassified event

dish

was

her

kitc

hen

fauc

et

detrended

derivative

bandpass

derivative

cepstrum

55%

85%

65%

90%

detrended

derivative

bandpass

derivative

cepstrum

signal feature comparisons

toile

t

unclassified event

term(i) template matching

compare across multiple signal transformations

dish

was

her

kitc

hen

fa

ucet

show

er

event library

unclassified event event library

toile

t

unclassified event

dish

was

her

kitc

hen

fauc

etshow

er

sig

nal

tran

sform

s

toile

tdi

shw

ashe

r

kitc

hen

fauc

etshow

er

unclassified event event library

unclassified eventassess

similarity

15%

55%

18%

9%

sig

nal

tran

sform

s

sig

nal

tran

sform

s

term(i) template matchingfilter transforms

term(i) template matchingtransforms: const-Q

unclassified event event library

toile

t

unclassified event

dish

was

her

kitc

hen

fauc

et

show

er

detrended

derivative

bandpass

derivative

const-Q cepstrum

sig

nal

tran

sform

s

unclassified event event library

toile

t

unclassified event

dish

was

her

kitc

hen

fauc

et

detrended

derivative

bandpass

derivative

const-Q cepstrum

55%

85%

65%

90%

detrended

derivative

bandpass

derivative

const-Q cepstrum

sig

nal

tran

sform

s

term(i) signal features

shower

toilet

bath

toilet

resonance tracking

unclassified event

template comparisons

signal feature comparisons

705030p

ress

ure

P(kitchen sink hot

open)

14%

P(kitchen sink cold

open)P(toilet open)

P(kitchen hot/cold

close)P(kitchen hot close)

P(toilet close)

3%

15%

2%

6%

1%

term (i): templates and signal features

705030p

ress

ure

P(kitchen sink hot

open)

14%

P(kitchen sink cold

open)P(toilet open)

P(kitchen hot/cold

close)P(kitchen hot close)

P(toilet close)

3%

15%

2%

6%

1%

term (i): templates and signal features

705030p

ress

ure

P(kitchen sink hot

open)

14%

P(kitchen sink cold

open)P(toilet open)

P(kitchen hot/cold

close)P(kitchen hot close)

P(toilet close)

3%

15%

2%

6%

1%

term (ii): bigram language model

705030p

ress

ure

P(kitchen sink hot

open)

14%

P(kitchen sink cold

open)P(toilet open)

P(kitchen hot/cold

close)P(kitchen hot close)

P(toilet close)

3%

15%

2%

6%

1%

term (ii): bigram language model

705030p

ress

ure

P(kitchen sink hot

open)

14%

P(kitchen sink cold

open)P(toilet open)

P(kitchen hot/cold

close)P(kitchen hot close)

P(toilet close)

3%

15%

2%

6%

1%

term (ii): bigram language model

705030p

ress

ure

P(kitchen sink hot

open)

14%

P(kitchen sink cold

open)P(toilet open)

P(kitchen hot/cold

close)P(kitchen hot close)

P(toilet close)

3%

15%

2%

6%

1%

4.6%

4.3%

4.1%

term (ii): bigram language model

sequence 1

sequence 2

sequence 3

705030p

ress

ure

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

kitchen sink hot open

shower cold open

toiletopen

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

sequence 1

sequence 2

sequence 3

term (ii): bigram language model

term(iii): grammar

opened closedopened closedtemperature consistency

1

2

3

soft penalty

705030p

ress

ure

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

bathroom sink hot open

shower cold open

toiletopen

bathroom sinkhot close

showercold close

kitchen sink

hot open

toiletclose

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

term(iii): grammar

sequence 1

sequence 2

sequence 3

705030p

ress

ure

bathroom sink hot open

shower cold open

toiletopen

bathroom sinkhot close

showercold close

kitchen sink

hot open

toiletclose

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

term(iii): grammar

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

705030p

ress

ure

bathroom sink hot open

shower cold open

toiletopen

bathroom sinkhot close

showercold close

kitchen sink

hot open

toiletclose

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

term(iii): grammar

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

705030p

ress

ure

bathroom sink hot open

shower cold open

toiletopen

bathroom sinkhot close

showercold close

kitchen sink

hot open

toiletclose

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

term(iii): grammar

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

fixture usage duration flow volume

term(iv): paired valve priors

0 20 40 6050

60

70

estimated flow volume

30 15060 90 120seconds

bin

fre

quency

3 96estimated gallons

1 12

open close

faucettoilet

bath faucet

toilet

bath

paired

705030p

ress

ure

term(iv): paired valve priors

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

bathroom sink hot open

shower cold open

toiletopen

bathroom sinkhot close

showercold close

kitchen sink

hot open

toiletclose

kitchen sinkhot close

705030p

ress

ure

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

bathroom sink hot open

shower cold open

toiletopen

bathroom sinkhot close

showercold close

kitchen sink

hot open

toiletclose

kitchen sinkhot close

term(iv): paired valve priors

705030p

ress

ure

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

term(iv): paired valve priorstoilet

kitchen sink kitchen sinkbathroom

sink

three levels of granularity

valve levele.g., upstairs bathroom faucet hot water activated

1

fixture levele.g., upstairs bathroom faucet activated

2

fixture category levele.g., faucet activated

3

hydrosense classification resultsreal-world water usage data

0%

20%

40%

60%

80%

100%

75.5% 89.5% 95.9%

one sensor, terms(i)-(iv)

fixture category

fixturevalve

*error bars = std error *10-fold cross validation, 15000 events

0%

20%

40%

60%

80%

100%

75.5% 89.5% 95.9%82.4% 93.5% 97.7%

hydrosense classification resultsreal-world water usage data

two sensors, terms(i)-(iv)

fixture category

fixturevalve

*error bars = std error

one sensor, terms(i)-(iv)

*10-fold cross validation, 15000 events

705030pre

ssu

re

kitchen sink hot open

kitchen sink cold open

dishwasher open

bathroom sinkhot close

kitchen sinkhot close

toiletopen

toiletclose

kitchen sinkhot close

kitchen sink hot open

toiletopen

bathroom sinkhot open

bathroom sinkhot close

kitchen sinkhot close

kitchen sink

hot open

kitchen sinkhot close

toilet close

bathroom sink hot open

shower cold open

toiletopen

bathroom sinkhot close

showercold close

kitchen sink

hot open

toiletclose

kitchen sinkhot close

hand washing

brushing teeth

washing dishes

shaving… …

TED for water

water fixture

classification

health sensing

from phone

user interface sensing

counting coughs

Self Report Sensing

annotate each sound type

data collectionGo back to daily routine for 3-7 hours

pay attention to your cough frequency

Come back and self-report cough frequencydata annotation

One week pilot and set up guideline and shared wiki

4 weeks6 linguistic students

cough speech laughter

breath

sneeze wind sniff noise

throat-clearing others’ cough

Come to Lab

Coughing?

PCA tree bagging decision

48000features

13features

future work• spirometry from a mobile phone• working with children’s hospital in seattle

mySpirometer App, 2009

TED for water

water fixture

classification

health sensing

from phone

user interface sensing

0.1um 1um 10 um 100 um

Wavelength

Visible InfraredUV

Sou

rce

Inte

nsi

ty

Solar Reflection

Thermal Radiation

RGBDept

hThermal

Near Far Infrared

strokes

chording

hand prints

curves

multi-user pressure

Jon FroehlichTim CampbellSean LiuSidhant Gupta

Eric SwansonTien-Jui LeeElliot SabaGabe Cohn

Where signal processing meets ubiquitous computing:

the role of DSP in the vision of UbiComp; Sustainability, Health, and

User InterfacesEric Larsoneclarson@uw.eduubicomplab.cs.wahington.edu@ericcooplarson