activity recognition using commonsense reasoningalumni.media.mit.edu/~emunguia/pdf/common sense...
TRANSCRIPT
![Page 1: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/1.jpg)
Activity Recognition Using Commonsense
Reasoning
Pallavi Kaushik and Emmanuel Munguia Tapia
![Page 2: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/2.jpg)
The pain
• The aging
–Potential loss of independence
–Fear
–Extraordinary costs
• The caregivers
–Difficult decisions
–Family disruption
–Extraordinary costs (money and time)
• Traditional medical system often of little help
![Page 3: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/3.jpg)
Limited options
• Aging parent moves in with family
– Tensions
– Space/renovations
– Cost
• Family moves in with aging parent
– Rarely practical
• Home care
• Assisted living
All options exacerbated by distance
![Page 4: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/4.jpg)
The extraordinary cost of care…Nursing
homes
Assisted
living
facilities
Independent
living
facilities
Home
care, 1
visit/day
Adult day
care
Emergency
response
service
Medication
reminder
service
Yearly $41,975 $13,140 -
$64,970
$13,140 -
$64,970
$31,025 $3,650 -
$18,250
$12,775 -
$18,250
$250
![Page 5: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/5.jpg)
One vision: ActivityLink
Providing peace of mind communication
![Page 6: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/6.jpg)
Gadgets for peace of mind?
![Page 7: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/7.jpg)
The single aging parent
![Page 8: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/8.jpg)
… and then neighbor concerns
• Neighbor tells adult son that she hasn’t
noticed his mother gardening lately, and son
sees that garden is untended
![Page 9: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/9.jpg)
Dignified piece of mind: a
standout
![Page 10: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/10.jpg)
ActivityLink available at home
stores
![Page 11: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/11.jpg)
Hundreds of sensors: a kit of
parts
Just a few hundred dollars and $20 a month
![Page 12: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/12.jpg)
Installation elder’s home
![Page 13: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/13.jpg)
Stick-ems easy, wireless …
everyone helps.
![Page 14: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/14.jpg)
Sensors installed in home of
adult children
![Page 15: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/15.jpg)
Devices custom & invisible
Install in just a few hours
![Page 16: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/16.jpg)
System models overall activity
Softwaredetects
“important”
(3-ring)changes
![Page 17: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/17.jpg)
Change detected
“Something’s changed … give her a call”
![Page 18: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/18.jpg)
System lifestyle benefits
• Conversations don’t always start with…
– “Are you still seeing friends?”
– “Are you still taking your medication?”
– “Are you still getting exercise?”
• Easy install, no maintenance
• Sensors “invisible”: not stigmatizing
• Batteries replaced casually, once a year
• Both households have system and use it
![Page 19: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/19.jpg)
Infrastructure enables other
applications
• Fun
• Communication
• New proactive applications for keeping people:
– Active
– Mindful
– Empowered
![Page 20: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/20.jpg)
Current work: proactive health
• Switch/bend sensors
– Doors
– Cabinets
– Drawers
– Thresholds
– Appliances
– Objects
• Wearable sensors
– Accelerometers
– Heart rate monitor
– Self report
• Multi-purpose sensors
– People-locator tags
– Auditory sensors
– Optical sensors
Activity recognition
Eating meals
Talking
Sleeping patterns
Taking medications
Cleaning
Cooking
…
newML algorithms
Detect change in activity;Motivate behavior changes
healthapplications
![Page 21: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/21.jpg)
Best bet: link advice with activity
• Simple messages
• Right time
• Right place
• Non-disruptive
• Big impact
– 20% shown for energy
– Substantial gains for preventative medicine
Requires computational sensing
Requires “pixels where you are”
Requires attention to UI design
![Page 22: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/22.jpg)
Objectives
• Include end-user in the system loop
• Use common sense to generate human
models of activity.
• Use these models as prior information in ML
algorithms to reduce number of training
examples required for classification.
• Use common sense reasoning to infer what
objects might be inside drawers and cabinets
![Page 23: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/23.jpg)
Common sense attributes
![Page 24: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/24.jpg)
Conventional Machine Learning
![Page 25: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/25.jpg)
Using Common sense
How does it work?
![Page 26: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/26.jpg)
GUI for generating the activity model
![Page 27: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/27.jpg)
Example models generated (OMICS)
Bathing
shower
spray
cleaner
sponge
water
brush
drain
agent
tub
wash
bathtub
Laundry
clothes
washing
machine
hanger
clothing
laundry
washer
fold
piece
closet
gather
Preparing dinner
pot
microwave
soup
stove
gas
pan
heat
table
Container
refrigerator
timer
(Objects used and probability)
![Page 28: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/28.jpg)
Using Google to extract time and
room information
![Page 29: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/29.jpg)
Room examples
![Page 30: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/30.jpg)
Time probabilities examples
![Page 31: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/31.jpg)
Model->NB classifierClass entering-the-house: Prior probability = 0.16
door: Discrete Estimator. Counts = 10 90 (Total = 100)
cabinet: Discrete Estimator. Counts = 95 5 (Total = 100)
drawer: Discrete Estimator. Counts = 95 5 (Total = 100)
couch: Discrete Estimator. Counts = 95 5 (Total = 100)
sofa: Discrete Estimator. Counts = 95 5 (Total = 100)
table: Discrete Estimator. Counts = 95 5 (Total = 100)
chair: Discrete Estimator. Counts = 95 5 (Total = 100)
light-switch: Discrete Estimator. Counts = 95 5 (Total = 100)
lamp: Discrete Estimator. Counts = 95 5 (Total = 100)
closet: Discrete Estimator. Counts = 95 5 (Total = 100)
window: Discrete Estimator. Counts = 10 90 (Total = 100)
faucet: Discrete Estimator. Counts = 95 5 (Total = 100)
stove: Discrete Estimator. Counts = 95 5 (Total = 100)
trash-can: Discrete Estimator. Counts = 95 5 (Total = 100)
![Page 32: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/32.jpg)
Looking around with Common sense
![Page 33: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/33.jpg)
Testing over real data
![Page 34: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/34.jpg)
System output examples
![Page 35: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/35.jpg)
12:00a 1:15a 2:31a 3:47a 5:03a 6:18a 7:34a 8:50a 10:06a 11:22a 12:37p 1:53p 3:09p 4:25p 5:41p 6:56p 8:12p 9:28p 10:44p 12:00p
92 Office/study Light-sw itch
82 Office/study Draw er
75 Bedroom Draw er
71 Bedroom Draw er
62 Bedroom Draw er
146 Bedroom Draw er
108 Bedroom Light-sw itch
95 Kitchen Light-sw itch
94 Kitchen Burner
91 Kitchen Refrigerator
84 Kitchen Draw er
80 Kitchen Cabinet
78 Kitchen Draw er
73 Kitchen Cabinet
72 Kitchen Cabinet
70 Kitchen Dishw asher
66 Kitchen Cabinet
55 Kitchen Cabinet
54 Kitchen Door
53 Kitchen Cabinet
143 Kitchen Microw ave
137 Kitchen Freezer
135 Kitchen Draw er
120 Kitchen Light-sw itch
105 Kitchen Light-sw itch
140 Foyer Door
104 Foyer Light-sw itch
93 Bathroom Show er-faucet
88 Bathroom Sink-faucet---cold
68 Bathroom Sink-faucet---hot
67 Bathroom Cabinet
58 Bathroom Medicine-cabinet
57 Bathroom Medicine-cabinet
130 Bathroom Door
101 Bathroom Light-sw itch
100 Bathroom Toilet-Flush-
Thursday, 3/27/2003
Thursday, 3/27/2003
![Page 36: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/36.jpg)
12:00a 1:15a 2:31a 3:47a 5:03a 6:18a 7:34a 8:50a 10:06a 11:22a 12:37p 1:53p 3:09p 4:25p 5:41p 6:56p 8:12p 9:28p 10:44p 12:00p
unknown
preparing lunch
preparing dinner
laundry
cleaning
bathing
entering the house
![Page 37: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/37.jpg)
What is difficult about testing the
models over real data?
• Need to map activity models to user activities
one-to-one
laundry->doing laundry
one-to-multiple
preparing meal->preparing breakfast
preparing meal->preparing lunch
preparing meal->preparing dinner
multiple-to-one
cleaning bathroom->cleaning
cleaning kitchen->cleaning
cleaning study->cleaning
How to do it automatically?
![Page 38: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/38.jpg)
What is difficult about testing the
models over real data?
• Need to map model objects to sensor objects
Sensor -> Model
Shower->Faucet
Stood->Chair
Dishwashing liquid->detergent
• Not all objects in house have sensors, particularly small ones
How to propagate probabilities robustly?
Drawer->Objects_Inside
![Page 39: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/39.jpg)
Things we have learned
• Assigning fixed probabilities to used objects is problematic. Improve probs from google or OMICS.
• Google probabilities for time and room are not great.
• Activities such as cleaning are problematic.
• Collapse breakfast, lunch and dinner into a single preparing meal class.
• String matching is a problem
• Similarity functions implemented need to be improved
![Page 40: Activity Recognition Using Commonsense Reasoningalumni.media.mit.edu/~emunguia/pdf/common sense activity... · 2016. 1. 4. · • Switch/bend sensors –Doors –Cabinets –Drawers](https://reader035.vdocuments.site/reader035/viewer/2022071105/5fdfb6aa3fc54c2c945c6a26/html5/thumbnails/40.jpg)
Future work
• Extract probability of objects from common sense
• Extract binary features over objects e.g. pot_stove, stove_fridge
• Experiment with other ways to extract room and time information
• Explore new ways to test the models and do the activities/objects mapping