activity recognition using commonsense reasoningalumni.media.mit.edu/~emunguia/pdf/common sense...
Post on 29-Aug-2020
4 Views
Preview:
TRANSCRIPT
Activity Recognition Using Commonsense
Reasoning
Pallavi Kaushik and Emmanuel Munguia Tapia
The pain
• The aging
–Potential loss of independence
–Fear
–Extraordinary costs
• The caregivers
–Difficult decisions
–Family disruption
–Extraordinary costs (money and time)
• Traditional medical system often of little help
Limited options
• Aging parent moves in with family
– Tensions
– Space/renovations
– Cost
• Family moves in with aging parent
– Rarely practical
• Home care
• Assisted living
All options exacerbated by distance
The extraordinary cost of care…Nursing
homes
Assisted
living
facilities
Independent
living
facilities
Home
care, 1
visit/day
Adult day
care
Emergency
response
service
Medication
reminder
service
Yearly $41,975 $13,140 -
$64,970
$13,140 -
$64,970
$31,025 $3,650 -
$18,250
$12,775 -
$18,250
$250
One vision: ActivityLink
Providing peace of mind communication
Gadgets for peace of mind?
The single aging parent
… and then neighbor concerns
• Neighbor tells adult son that she hasn’t
noticed his mother gardening lately, and son
sees that garden is untended
Dignified piece of mind: a
standout
ActivityLink available at home
stores
Hundreds of sensors: a kit of
parts
Just a few hundred dollars and $20 a month
Installation elder’s home
Stick-ems easy, wireless …
everyone helps.
Sensors installed in home of
adult children
Devices custom & invisible
Install in just a few hours
System models overall activity
Softwaredetects
“important”
(3-ring)changes
Change detected
“Something’s changed … give her a call”
System lifestyle benefits
• Conversations don’t always start with…
– “Are you still seeing friends?”
– “Are you still taking your medication?”
– “Are you still getting exercise?”
• Easy install, no maintenance
• Sensors “invisible”: not stigmatizing
• Batteries replaced casually, once a year
• Both households have system and use it
Infrastructure enables other
applications
• Fun
• Communication
• New proactive applications for keeping people:
– Active
– Mindful
– Empowered
Current work: proactive health
• Switch/bend sensors
– Doors
– Cabinets
– Drawers
– Thresholds
– Appliances
– Objects
• Wearable sensors
– Accelerometers
– Heart rate monitor
– Self report
• Multi-purpose sensors
– People-locator tags
– Auditory sensors
– Optical sensors
Activity recognition
Eating meals
Talking
Sleeping patterns
Taking medications
Cleaning
Cooking
…
newML algorithms
Detect change in activity;Motivate behavior changes
healthapplications
Best bet: link advice with activity
• Simple messages
• Right time
• Right place
• Non-disruptive
• Big impact
– 20% shown for energy
– Substantial gains for preventative medicine
Requires computational sensing
Requires “pixels where you are”
Requires attention to UI design
Objectives
• Include end-user in the system loop
• Use common sense to generate human
models of activity.
• Use these models as prior information in ML
algorithms to reduce number of training
examples required for classification.
• Use common sense reasoning to infer what
objects might be inside drawers and cabinets
Common sense attributes
Conventional Machine Learning
Using Common sense
How does it work?
GUI for generating the activity model
Example models generated (OMICS)
Bathing
shower
spray
cleaner
sponge
water
brush
drain
agent
tub
wash
bathtub
Laundry
clothes
washing
machine
hanger
clothing
laundry
washer
fold
piece
closet
gather
Preparing dinner
pot
microwave
soup
stove
gas
pan
heat
table
Container
refrigerator
timer
(Objects used and probability)
Using Google to extract time and
room information
Room examples
Time probabilities examples
Model->NB classifierClass entering-the-house: Prior probability = 0.16
door: Discrete Estimator. Counts = 10 90 (Total = 100)
cabinet: Discrete Estimator. Counts = 95 5 (Total = 100)
drawer: Discrete Estimator. Counts = 95 5 (Total = 100)
couch: Discrete Estimator. Counts = 95 5 (Total = 100)
sofa: Discrete Estimator. Counts = 95 5 (Total = 100)
table: Discrete Estimator. Counts = 95 5 (Total = 100)
chair: Discrete Estimator. Counts = 95 5 (Total = 100)
light-switch: Discrete Estimator. Counts = 95 5 (Total = 100)
lamp: Discrete Estimator. Counts = 95 5 (Total = 100)
closet: Discrete Estimator. Counts = 95 5 (Total = 100)
window: Discrete Estimator. Counts = 10 90 (Total = 100)
faucet: Discrete Estimator. Counts = 95 5 (Total = 100)
stove: Discrete Estimator. Counts = 95 5 (Total = 100)
trash-can: Discrete Estimator. Counts = 95 5 (Total = 100)
Looking around with Common sense
Testing over real data
System output examples
12:00a 1:15a 2:31a 3:47a 5:03a 6:18a 7:34a 8:50a 10:06a 11:22a 12:37p 1:53p 3:09p 4:25p 5:41p 6:56p 8:12p 9:28p 10:44p 12:00p
92 Office/study Light-sw itch
82 Office/study Draw er
75 Bedroom Draw er
71 Bedroom Draw er
62 Bedroom Draw er
146 Bedroom Draw er
108 Bedroom Light-sw itch
95 Kitchen Light-sw itch
94 Kitchen Burner
91 Kitchen Refrigerator
84 Kitchen Draw er
80 Kitchen Cabinet
78 Kitchen Draw er
73 Kitchen Cabinet
72 Kitchen Cabinet
70 Kitchen Dishw asher
66 Kitchen Cabinet
55 Kitchen Cabinet
54 Kitchen Door
53 Kitchen Cabinet
143 Kitchen Microw ave
137 Kitchen Freezer
135 Kitchen Draw er
120 Kitchen Light-sw itch
105 Kitchen Light-sw itch
140 Foyer Door
104 Foyer Light-sw itch
93 Bathroom Show er-faucet
88 Bathroom Sink-faucet---cold
68 Bathroom Sink-faucet---hot
67 Bathroom Cabinet
58 Bathroom Medicine-cabinet
57 Bathroom Medicine-cabinet
130 Bathroom Door
101 Bathroom Light-sw itch
100 Bathroom Toilet-Flush-
Thursday, 3/27/2003
Thursday, 3/27/2003
12:00a 1:15a 2:31a 3:47a 5:03a 6:18a 7:34a 8:50a 10:06a 11:22a 12:37p 1:53p 3:09p 4:25p 5:41p 6:56p 8:12p 9:28p 10:44p 12:00p
unknown
preparing lunch
preparing dinner
laundry
cleaning
bathing
entering the house
What is difficult about testing the
models over real data?
• Need to map activity models to user activities
one-to-one
laundry->doing laundry
one-to-multiple
preparing meal->preparing breakfast
preparing meal->preparing lunch
preparing meal->preparing dinner
multiple-to-one
cleaning bathroom->cleaning
cleaning kitchen->cleaning
cleaning study->cleaning
How to do it automatically?
What is difficult about testing the
models over real data?
• Need to map model objects to sensor objects
Sensor -> Model
Shower->Faucet
Stood->Chair
Dishwashing liquid->detergent
• Not all objects in house have sensors, particularly small ones
How to propagate probabilities robustly?
Drawer->Objects_Inside
Things we have learned
• Assigning fixed probabilities to used objects is problematic. Improve probs from google or OMICS.
• Google probabilities for time and room are not great.
• Activities such as cleaning are problematic.
• Collapse breakfast, lunch and dinner into a single preparing meal class.
• String matching is a problem
• Similarity functions implemented need to be improved
Future work
• Extract probability of objects from common sense
• Extract binary features over objects e.g. pot_stove, stove_fridge
• Experiment with other ways to extract room and time information
• Explore new ways to test the models and do the activities/objects mapping
top related