machine learning family routines

Scott DavidoffDissertation Defense

FAMILY CONTROL SMART HOMETHE PROJECT ON

Routine as Resource for the Design

of Learning Systems

Scott Davidoff Dissertation Defense


A pattern of behavior that is followed

repeatedly, but is subject to change

if conditions change

Winter 1964

“

“”

1


Koestler 1967

Routines reduce our attention needs

2


Zerubavel 1981

Reduce attention needed for tasks

3


Wakkary & Maestri 2007

Free attention for bigger challenges

4


Wolin + Bennett 1984

5


The cyclic nature of routine makes it

a natural target for machine learning


Machine learning can use this order

7


Suchman 1983, Tolmie et al. 2002

Idiosyncrasies are hard to model

8


Certain human routines can be

modeled, increasing the scope of

activity recognition

Machine learning

Opportunities

9


The ability to use learned routines in

end-user applications would solve a

variety of human problems

HCI

Opportunities

10


Dual-income family logistics

11


Sequence of place and

transportation (rides) that occur on

daily, weekly, and seasonal cycles

“

“”

12


Darrah et al. 2000

Managing details can be difficult

13


Frissen 2000

Routines give a feeling of control

14


Beech et al. 2004

Life does not always follow routines

15


Breakdowns lead to loss of control

16


Perry et al. 2001, Ling + Campbell 2003

A constant need to follow updates

17


Gneezy + Rustichini 1998, Darrah 2009

A constant source of anxiety

18


We can learn a model of family

logistical routines, and present that

information to families to help them

feel more in control of their lives

19

Scott DavidoffDissertation Defense 20


2We can use sensing

and modeling to

synthesize missing

information resources

3Show how to use the

model and evaluate the

impact of the

information

1We can use fieldwork to

identify missing but

needed information

resources

Fieldwork Modeling Validation

22


Create an ordered list of places and rides

Attend to the details of a plan as it unfolds

Planning and coordinationLogistics

Coordinate

Plan

23


Fieldwork1

24


1805

22 Davidoff et al Ubicomp 2006Months

Months

Months0606

12Families

Families

Families

Davidoff et al Ubicomp 2007

Davidoff, Dey + Zimmerman CHI 2009

45 Months24Families

25


Semi-Structured Interviews

Davidoff et al. Ubicomp 2006

26

Scott DavidoffDissertation Defense scott davidoff, min kyung lee, anind dey + john zimmerman

Needs Validation



User Enactments

28


User Enactments


29


GPS

Phone Calls

Email

SMS

Calendars


30


528 phone interviews

109 activity Interviews

108 calendar months


31


Less than 20% of days go exactly as

planned1

Routines are not documented

People have incomplete knowledge

of other people’s routines

People make plans that depend on

incorrect information

2

3

4

32


Family E

Routine

Deviation, Scheduled

Deviation, Unscheduled

34


Family EAll Families

Routine

Deviation, Scheduled

Deviation, Unscheduled

35


Less than 20% exactly as planned

36


Routineness of Family E Activities

37



38



39



Fieldwork Finding


40



August September October November

41


People have incomplete knowledge

of other people’s routines

Fieldwork Finding


42



43



44



45



46



47



48



49



50


People make plans that depend on

incorrect information

Fieldwork Finding


51


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: Dad

Check-up

Track Practice

52


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: S16

Paper Route

Track Practice

53


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Who Will do the Paper Route?

Check-up

Track Practice

Paper Route

54


Information gaps can break down

coordination

No resources exist to find needed

information


55


Modeling2

56


2 31Ride

Detection

Driver

Prediction

Predict

Lateness

57


In Situ

Observation

Maintain Current

Behaviors

Ubiquitous

Sensing

GPS

58


Parent drives kid to an activity

Parent drives kidfromactivity

Pick-ups and drop-offsRide

Pick-up

Drop-off

59


Ride Detection


Day Care

Work

Using GPS to Sense a Drop-Off

Parent Child 61

t3 t4 t5 t6t2t1 t7


Day Care

Work


Parent Child 62

t3 t4 t5 t6t2t1 t7


Day Care

Work


Parent Child 63

t3 t4 t5 t6t2t1 t7


t3Day Care

t4 t5 t6t2

Work

t1 t7


Parent Child 64


Day Care

Work

Using GPS to Sense a Pick-Up

Parent Child 65

t3 t4 t5 t6t2t1 t7


Day Care

Work


Parent Child

t3 t4 t5 t6t2t1 t7


GPS Ride Detection Evaluation

Family Precision Recall

How many of the sensed rides

are right

How many of the total rides

are sensed?

Pick-up Drop-off Pick-up Drop-off

A .991 .987 .912 .910

B .966 .962 .979 .981

C .913 .824 .971 .921

D .878 .873 .980 .944

E .931 .684 .959 .985

Mean .936 .866 .960 .950

.901 .955

68


Smarter power management

Wi-Fi, Bluetooth, cell tower ID

Apply heuristics to modeling

Cultural norms, Individual Behavior

Single Location Sensor

GPS Sampling Rate

69


Driver Prediction


Name Meaning Values

Ln Location of pick-up or drop-off Place ID

RType Ride type Pick-up, Drop-off

DoW Day of week 0,1,2,3,4,5,6

ToD Discretized time of day (15 min) 1,2,3…96

drivert-j Driver for the last 5 rides to Ln Mom, Dad

Driver distribution model [0,1]

Driver Prediction Model Features

71


Driver Prediction Performance x

Number of Training Weeks of Data

1 24 1 24 1 24 1 24 1 24 1 24 1 24

1 24 1 24 1 24 1 24 1 24 1 24 1 24

72




1 24 1 24 1 24 1 24 1 24 1 24 1 24

1 24 1 24 1 24 1 24 1 24 1 24 1 24

73




1 24 1 24 1 24 1 24 1 24 1 24 1 24

1 24 1 24 1 24 1 24 1 24 1 24 1 24

74


Lateness Prediction


TnowTideal

Lateness in Minutes

N

u

m

b

e

r

o

f

E

x

a

m

p

l

e

s

Distribution of Parent Late Arrivals

N = 83


Name Meaning Values

R Whether the parent

remembers

True, False

Lchild Location of the child Place ID

Lcurr Current location of a

parent

Place ID

J Driver prediction model Mom, Dad

77


Name Meaning Values


remembers

True, False



parent

Place ID


T If the parent is traveling True, False

Lstart Starting location of a

parent

Place ID

78


Name Meaning Values


remembers

True, False



parent

Place ID




parent

Place ID

Empirical cumulative

distribution (ecdf)of

on-time arrivals to Lchild

at time TnowTideal

[0,1]

79


Name Meaning Values


remembers

True, False



parent

Place ID




parent

Place ID




at time TnowTideal

[0,1]

D Destination of a parent Place ID

80


Name Meaning Values


remembers

True, False



parent

Place ID




parent

Place ID




at time TnowTideal

[0,1]

D Destination of a parent Place ID

81


tideal- 30

A’ = 0.659

.6

.5


tideal

A’ = 0.801


tideal+10

A’ = 0.826






85


Validation3

86


Do parents understand the view?

Do parents perceive that the

information is valuable?

Do parents feel more in control

having this view?

88


1257

45Davidoff et al Ubicomp 2006

Davidoff et al Ubicomp 2007


Davidoff et al. CHI 2010Months

Months24

24Families

Families

Families


(in preparation)

89


Davidoff et al. Ubicomp 2011 in preparation

Experience prototype: Doctor

90


Davidoff et al. Ubicomp 2011 in preparation

Experience prototype: Kitchen

93


It's nice to have a map of the day. I

don't literally write all this stuff down,

so sometimes it's just hard to keep

in my head.

Visual Distills Details

“

94

−P2“”


Because the calendar [is] all words

and numbers, so you have to really

think about …how long everything

takes…

Visual Simplifies Calculation

“

95

−P10


…With the visual you can see who's

doing what at what time. You can

make a decision about who's

activity to change.

Visual Simplifies Calculation

96

−P10“”


I would be able to know where

everyone was gonna be, instead of

having to ask around.

Visual Clarifies Intentions

“”97

−P11

“


Venkatesh + Davis 2003

Technology Acceptance Model-3

98


…helps me do my job “as a parent.”

…would want my family to use it.

…more in control of details, more

ready for changes

Perception of Control

Behavioral Intention

Perceived Usefulness

99


Perceived

Usefulness

Behavioral

Intention

Perception of

Control

Supports

Planning

Supports

Awareness

7

6

5

4

3

2

1






101


Contributions

102


Suchman 1983, Tolmie et al. 2002

Certain routines can be modeled

103


2 31Ride

Detection

Driver

Prediction

Predict

Lateness

104


Commodity GPS with

real power constraints

105


Breakdowns lead to loss of control

107


Perceived

Usefulness

Behavioral

Intention

Perception of

Control

Supports

Planning

Supports

Awareness

7

6

5

4

3

2

1




109


Empower Parents


110


Future Work

111


Improve algorithms

Use algorithms to enable other

family coordination systems

Add additional sensor inputs

Extend to other forms of routine

112

Scott DavidoffDissertation Defense Neustaedter + Brush CHI 2006 113


6pm

Orthodontist

Neustaedter + Brush CHI 2006 114


6pm

Ortho

Paper Route

Neustaedter + Brush CHI 2006 115

Scott DavidoffDissertation Defense Brown et al. Ubicomp 2007 116

Scott DavidoffDissertation Defense Brown et al. Ubicomp 2007 117


FAMILY CONTROL SMART HOME

The project on


Thanks for your time, brain power, and support


Thanks to our sponsors

NSF IIS 1017429

Google Research Award


Thanks to our supporters


Thanks to my collaborators


Thanks to my committee


Thanks to my advisors


Questions


Image Creditshttp://thesituationist.files.wordpress.com/2008/06/traffic.jpg

http://kaleidoscope.cultural-

china.com/chinaWH/upload/upfiles/2009-

06/22/unique_morning_exercises_a_daily_routine_for_countles

s_chinese67f17841b1f4bfae458c.jpg

http://travellingcam.files.wordpress.com/2006/11/japan-self-

defense-force-marching-festival-2006-7.jpg

http://thhsmusic.com/Images/2007_Marching_Unit.jpg

http://0.tqn.com/d/golosangeles/1/0/-/H/-/-/2007-juggling3.jpg

http://popten.net/wp-content/uploads/2009/03/traffic-jam.jpg

http://www.lateott.com/images/control_key.jpg

http://stuffunemployedpeoplelike.com/wp-

content/uploads/2009/05/carpool.jpg

http://larchmontgazette.com/commentary/columns/soccermom.j

pg

http://www.schoolbrief.com/SchoolBrief/images/parent.jpg

http://kmcavoy.edublogs.org/files/2010/11/6a00d83451b3c669e

200e54f94c4bb8833-500wi-xl20xk.jpg

http://www.weshipyourcar.com/LinkPageImages/early-flight.gif

http://memory.loc.gov/master/mss/mwright/04/04003/0030d.jpg

http://www.mnn.com/sites/default/files/user-

60/school%20bus_0.jpg

http://arlingtonkids.files.wordpress.com/2011/01/school-bus.jpg

http://4.bp.blogspot.com/_K-

aTC4UHVR0/S7tDj1UfVSI/AAAAAAAAAB4/QTg9MTxuecY/s16

00/confused.jpg

http://cdn.sheknows.com/articles/10-Qualities-SM-Art-Img.jpg

http://i.dailymail.co.uk/i/pix/2009/10/25/article-0-

059F2EC8000005DC-613_468x329.jpg

http://static.tvguide.com/MediaBin/Galleries/Editorial/090511/fat

guys/fatguys-honeymooners8.jpg

http://www.ministryofmanipulation.com/images/8bcube.jpg

http://http//vandoverviewpoints.com/wp-

content/themes/Yamidoo/yamidoo/scripts/timthumb.php?src=htt

p://vandoverviewpoints.com/wp-

content/uploads/2010/04/Family_back.jpg&w=390&h=600&zc=

1

http://www.wmich.edu/registrar/assets/images/photos/calendar2

.jpg

http://www.mediabistro.com/fishbowlDC/files/original/1924_chec

king_facts.jpg

http://www.sbac.edu/~media/images/questionmark.jpg

http://img.dailymail.co.uk/i/pix/2008/02_04/textDM2502_468x35

2.jpg

http://farm3.static.flickr.com/2387/2137448527_eb889a39cb.jpg

http://www.zoodles.com/blog/wp-content/uploads/2010/08/lost-

child4-300x225.jpg

126


QUESTION

SLIDES

127


ROUTINE

MODELING

RELATED WORK

128


FORMULAS

129


Driver Distribution

130


LWDT

131


Bayesian Network

132


RIDE SENSING

TECHNICAL

VERSION

133


Davidoff, Dey& Zimmerman CHI 2011

134


Two Kinds of Drop-Offs

135


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

136


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

137


Two Kinds of Pick-Ups

138


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

139


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

140


5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


141


5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


142


5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


143


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


144


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


145


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


146


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


147


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


148


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


149


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


150


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


151


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


152


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


153


SCHOOL HALF

DAY

BREAKDOWN

154


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: S16

Track Practice

155


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: S16

Track Practice

156


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Dad Started Baseball League

Track Practice

157


OTHER

APPLICATIONS

158


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Detect conflict with unlabeled routine

Track Practice

Check-up

Paper Route

159


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Dad Started Baseball League

Track Practice

160


ACTIVITY

SCATTERPLOT

161


RIDE SENSING

ERRORS

162


Smarter power managementGPS Sampling Rate

163



Wi-Fi, cell tower, bluetooth


GPS Sampling Rate

164


5:15

Home

5:30 5:45 6:005:00

Work

4:45 6:15

Simultaneous Departure Errors

School

Parent Child165



Wi-Fi, cell tower, bluetooth

Apply heuristics to modeling

Cultural norms, Individual Behavior


GPS Sampling Rate

166


1:00pm

Home

2:00 3:00 4:0012:00

Work

11:00 5:00

Lost Signal During Daytime

School

Parent Child167


DRIVER

PREDICTION

ERRORS

168




169


USER

ENACTMENT

PICS

170


User Enactments


171


User Enactments


172


UNUSED PICS

173


Koestler 1967


174


Darrah et al. 2000, Frissen 2000, Beech et al. 2004

People can’t see all the facts

175



Managing Rides Causes Anxiety

176


Pentland + Reuter 1994

Graceful

177


Wolin& Bennett 1984

Path for exercise

178


Medved 2004

Confident

179


HCI Approach to Machine Learning

180


Dual-income families

181


Frissen 2000, Fiese et al. 2002, Davidoff et al. 2007

Routines help parents feel in control

182


Frissen 2000, Fiese et al. 2002, Davidoff et al. 2007

Just surviving the day

183


Darrah et al. 2000, Frissen 2000, Beech et al. 2004

Non-Routine Events Cause Anxiety

184




185




186


How do routines provide support?


187


LONG-FORM

APPROACH

188


We can use fieldwork to identify

missing but needed information

resources

Fieldwork

189

1


We can use sensing and modeling

to synthesize missing information

resources

Modeling

190

2


We can develop an application to

deliver synthesized resources and

evaluate how it helps coordination

Show how to use the model and

validate that the application

potentially solves the problem by

Build and validation

191

3


We can exploit routine as a

resource to design learning systems

Thesis Statement

192


Understanding how routines support

daily life generate new design ideas

Generative Resource

193


Model information about family

transportation routines to

supplement family knowledge

Information Resource

194


We can exploit routine as a

resource to design learning systems

Thesis Statement

195


context-aware, adaptive, intelligent,

cognitive, expert, decision-support

196

system