m oodscope : s ensing mood from smartphone usage patterns
DESCRIPTION
Robert Likamwa Lin Zhong. M oodScope : S ensing mood from smartphone usage patterns. Asia. Yunxin Liu Nicholas D. Lane. Earlier today…. Mood-Enhanced Apps. Some time in the future…. Personal analytics. Social ecosystems. Media recommendation. Affective Computing (Mood and Emotion). - PowerPoint PPT PresentationTRANSCRIPT
MoodScope:Sensing mood from smartphone usage patterns
Robert LikamwaLin Zhong
Yunxin LiuNicholas D. Lane
Asia
2
Earlier today…
Mood-Enhanced Apps
Socialecosystems
Media recommendation
Personalanalytics
3 of 30
Some time in the future…
Affective Computing(Mood and Emotion)
Audio/Video-based(AffectAura, EmotionSense)
Biometric-based(Skin conductivity,
Temperature, Pulse rate)Highly temporalHigh cost of deploymentHassle
Captures expressionsPower hungrySlightly invasive
4 of 30
Can your mobile phone infer your
mood?
Can your mobile phone infer your
mood?From already-
available, low-power information?*
* No audio/video sensing, no body-instrumentation
MoodScope ∈ Affective Computing
Audio/Video-based
Usage Trace-based(MoodScope)
Biometric-based
Very direct, Fine-grainedHigh cost of deployment
Captures expressionsPower hungrySlightly invasive
Passive, ContinuousHow to model mood?
7 of 30
Mood is…• … a persistent long-lasting state
o Lasts hours or dayso Emotion lasts seconds or minutes
• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns
8 of 30
happysadnervousdepressed excited
relaxed
calm
stressed bored
Circumplex model (Russell 1980)
attentive
9 of 30
Mood is…• … a persistent long-lasting state
o Lasts hours or dayso Emotion lasts seconds or minutes
• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns
10 of 32
• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns
How is the user communicating?
What apps is the user using?
f ( ) = moodusage
13 of 30
iPhone Livelab Logger• Web history• Phone call history• Sms history• Email history• Location history• App usage
Adapted From C. Shepard, A. Rahmati, C. Tossel, L. Zhong, And P. Kortum, "Livelab: Measuring Wireless Networks And Smartphone Users In The Field," In Hotmetrics, 2010.
14 of 30
iPhone Livelab Logger• Web history• Phone call history• Sms history• Email history• Location history• App usage
Runs as shellHash private dataUploads logs to our server nightly
How can we generate mood labels?15 of 30
Mood Journaling App
User-base32 users aged between 18 and 29
11 females 16 of 30
Inference
• Detect a mood pattern
• Validate with only 60 days of data
• Wide range of candidate usage data
• Low computational resources
17 of 30
Daily Mood Averages
• Separate pleasure, activeness dimension
• Take the average over a day _______________
4
Σ( )
18 of 30
Exploring Features• Communication
o SMSo Emailo Phone Calls
• To whom?o# messageso Length/Duration
Consider “Top 10” Histograms
How many phone calls were made to #1? #2? … #10?
How much time was spent on calls to #1? #2? … #10?
19 of 30
?
?
Exploring Features• Communication
o SMSo Emailo Phone Calls
• To whom?o# messageso Length/Duration
• Usage ActivityoApplicationsoWebsites visitedo Location History
• Which (app/site/location)?o# instances
20 of 30
Previous Mood• Use previous 2 pairs of mood labels
21 of 30
Data Type Histogram by: Dimensions
Email contacts# Messages 10# Characters 10
SMS contacts# Messages 10# Characters 10
Phone call contacts# Calls 10Call Duration 10
Website domains # Visits 10Location Clusters # Visits 10
Apps# App launches 10App Duration 10
Categories of Apps# App launches 12
App Duration 12Previous Pleasure and Activeness Averages
N/A 4
22 of 30
??
• Multi-Linear Regressiono Minimize Mean Squared Error
• Leave-One-Out Cross-Validation
• Sequential ForwardFeature Selection during training
23 of 30
Model Design
Sequential Feature Selection
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Improvement of model as SFS adds more
features
Number of Features Used
Mean
Sq
uare
d E
rror (Each line is
a different user)
24 of 30
1 3 5 7 9 11 13 15 17 19 21 23 25 27 290
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Improvement of model as SFS adds more
features
Number of Features Used
Mean
Sq
uare
d E
rror
Sample Prediction
0 10 20 30 40 50 602
3
4
Mood (Pleasure)Estimated Mood
Days
Dail
y M
ood
Avera
ge
25 of 30
Error distributions• Error2 of > 0.25 will
misclassify a mood label
93% < 0.25 error2
0.0001
0.001
0.01
0.1
1
25%ile
10%ile
Users
Sq
uare
d E
rror
26 of 30
vs. Strawman ModelsModels using full-knowledge of a user’s data with LOOCV
Model A: Assume User’s Average Mood73% Accuracy
Model B: Assume User’s Previous Mood61% Accuracy
MoodScope Training: 93% Accuracy.
27 of 30
Personalized Training
10 20 30 40 50 590%
20%
40%
60%
80%
100%
Incremental personalized model
Training Days
Mod
el
Accu
racy
28 of 30
All-user modelaccuracy
Personalized/All-userHybrid Training
10 20 30 40 50 590%
20%
40%
60%
80%
100%
Incremental personalized model
Hybrid mood model
Training Days
Mod
el
Accu
racy
29 of 30
Phone
Mood Inputs/Usage Logs
Mood and Usage History
Cloud
Mood ModelMood Model
Current Usage Model
Training
Inferred Mood
Resource-friendly Implementation
30 of 30
Inferred Mood
API
TEXAS MEDICAL CENTER
RICE UNIVERSITY
Inferred Mood
API
MoodScope:Sensing mood from smartphone usage
patterns
• Robustly (93%) detect each dimension of daily moodo On personalized modelso Starts out with 66% on generalized
models
• Validate with 32 users x 2 months worth of data
• Simple resource-friendly implementation
35
Discriminative Features
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Improvement of model as SFS adds more
features
Number of Features Used
Mean
Sq
uare
d E
rror
Relevant features
36
Discriminative Features
Calls
Emai
lSM
SW
ebApp
s
Loca
...
Prev
....
0
20
40
60
80
100
120 Pleasure
Activeness
Nu
mb
er
of
Featu
res
37
TODO• Wider, longer-term evaluation
o How does the model change over time?
• In-use accuracy metricso Not cross-validation
• Social Factors/Impacto Study Mood-sharingo Provide assistance to psychologists