prefminer:)mining)user’s)preferences)for ...axm514/paper/slides/ubicomp16.pdf ·...

PrefMiner: Mining User’s Preferences for Intelligent Mobile No5fica5on Management

Abhinav Mehrotra University College London University of Birmingham

Robert Hendley University of Birmingham

Mirco Musolesi University College London

No5fica5ons Inform Users About a Variety of Events

System, Tools and Others

Online Social Networks Communica5on

Are All No5fica5ons Useful?

Are All No5fica5ons Useful?

NO

Users are not interested in many no5fica5ons they receive and feel annoyed on receiving them.

Interrup5bility Management System: State-‐of-‐the-‐art

Predict opportune moments by using:

•  context of the user. •  content of no5fica5on.

Limita8on No past study has considered stopping or detec5ng

no5fica5ons that are not useful for the user.

Interrup5bility Management System: State-‐of-‐the-‐art

Evaluated in the offline sePngs.

Limita8on Never evaluated in the real-‐world scenario.

Contribu5ons

1.  A mechanism for learning the types of informa5on users prefer to receive via no5fica5ons in different situa5ons.

2.  A technique to make the predic5ve model transparent to

users.

Mining User Preferences


Construc5ng Associa5on Rules

Removing Reminder

No5fica5ons

No5fica5on Clustering

If an applica5on’s click rate is zero then all

no5fica5ons from that applica5on are treated as reminder no5fica5ons.

Click Rate Number of accepted no5fica5ons Total number of no5fica5ons *

Some no5fica5ons are dismissed because they do not require any further ac5on from the user.

= 100

S1: Removing Reminder No5fica5ons



Removing Reminder

No5fica5ons


S2: No5fica5on Clustering

1.  Cleaning No5fica5on Titles •  Conversion of the text to lower-‐case •  Removal of punctua5on, numbers and stop words •  Removal of the sender and applica5on names •  Stemming of words

2.  Clustering No5fica5ons •  Removing sparse terms •  Term Frequency-‐based clustering

No5fica5on Title Applica5on Name

S2: No5fica5on Clustering

Removing sparse terms

TFthreshold Number of par5cipa5on days N * Total number of no5fica5ons =

This ensures that at least one no5fica5on containing the term is triggered in N days.

0

1000

2000

3000

1 2 3 4 5 6 7N−value

Term

s C

ount

Filtered Remaining Not much difference in the number of filtered terms with N ∈ [2, 7]. We use N = 2 for Tfthreshold .



Removing Reminder

No5fica5ons


S3: Construc5ng Associa5on Rules We use the AIS algorithm for mining associa5on rules.

X → Y

X is the antecedent (never contains no5fica5on response) Y as the consequent (only contains no5fica5on response)


X → Y


Support: Number of no5fica5ons of the no5fica5on-‐type covered in the rule

Total number of no5fica5ons in the dataset Confidence:

Occurrence count of the no5fica5on pabern covered in the rule Occurrence count of the no5fica5on pabern covered in the antecedent of rule

* 100

* 100


X → Y


{N1} → {Dismiss} {N2,Home} → {Accept} {N2,Other} → {Accept} {N2,Work} → {Dismiss}

Let us consider an example where the user: •  always dismiss N1 no5fica5ons. •  does not accept N2 no5fica5ons at work.

Procedure

1.  Evaluate (offline) the proposed intelligent no5fica5on mechanism. 2.  Funnel the findings in an intelligent no5fica5on library: MyPref. 3.  Implement an app on top of MyPref and deploy it to evaluate the

library in a real-‐world scenario.

Evalua5on (Offline)

Dataset

Study: My Phone and Me

Par5cipants: 18

Minimum ac5ve days: 14

No5fica5on samples: 11,185

Abhinav Mehrotra, Veljko Pejovic, Jo Vermeulen, Robert Hendley, and Mirco Musolesi. 2016. My Phone and Me: Understanding People’s Recep5vity to Mobile No5fica5ons. In CHI’16.

Types of Associa5on Rules

Assoc Rule 1: no5fica5on response with no5fica5on type. Assoc Rule 2: no5fica5on response with no5fica5on type and ac5vity. Assoc Rule 3: no5fica5on response with no5fica5on type and arrival 5me. Assoc Rule 4: no5fica5on response with no5fica5on type and loca5on. Assoc Rule 5: no5fica5on response with no5fica5on type, ac5vity, arrival 5me and loca5on.

Results

Recall: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are actually dismissed. Precision: ra5o between the number of no5fica5ons that are correctly predicted as dismissed and the total number of no5fica5ons that are predicted as dismissed.

0102030405060708090

100

70 80 90Confidence Value (%)

Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

AR4 AR5

Minimum Confidence: 65-‐95% Minimum Support: Dpar5cipate / ( 2 ∗ Ncount )

Dpar5cipate: Days of par>cipa>on Ncount: number of no5fica5ons collected for a user This ensures that the type of no5fica5on covered in each rule has arrived at least once in two days.

Assoc Rule 1 Assoc Rule 2 Assoc Rule 3

Results


0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

Assoc Rule 4 Assoc Rule 5


Results


0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

Best Performance


Assoc Rule 4 Assoc Rule 5

Results


0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

AR4 AR5

Best Performance

User’s preference for receiving a no8fica8on depends only on the type of informa8on it contains

and the loca8on.


Op5mizing the System for High Precision

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

Assoc Rule 4

•  We should aim to have fewer false-‐nega5ves (i.e., incorrectly predic5ng a no5fica5on as non-‐interes5ng for the user).

•  This can be achieved by ensuring that the precision remains close to 100%.

Op5mizing the System for High Precision

0102030405060708090

100


Res

ult (

%)

PrecisionRecall

•  We should aim to have fewer false-‐nega5ves (i.e., incorrectly predic5ng a no5fica5on as non-‐interes5ng for the user).

•  This can be achieved by ensuring that the precision remains close to 100%.

90% precision 35% recall

Assoc Rule 4

Online Learning

● ●

●

●

●

●●

●

●

●

● ● ●

●●

● ●

●

● ●●

90

100

1 3 5 7 9 11 13 15 17 19 21Day

Prec

isio

n (%

) Confidence● 65%

70%75%80%85%90%95%

● ●

●

●

●

●●

●

●

●

●● ● ● ●

● ● ●● ● ●

0

10

20

30

40

1 3 5 7 9 11 13 15 17 19 21Day

Rec

all (

%)

Confidence● 65%

70%75%80%85%90%95%

MyPref Library An intelligent interrup5bility management library that can predict the type of no5fica5ons that users would prefer to

receive on their mobile phones in specific contexts.

PlaQorm Implemented for the Android OS.

Evalua8on

It takes not more than 61 seconds for mining associa5on rules from 1500 no5fica5ons for any features.

Released as an open source project

hbps://github.com/AbhinavMehrotra/PrefMiner

In-‐the-‐wild Evalua5on

PrefMiner

1 2 3 4

In-‐the-‐wild Evalua5on Results

Par5cipants: 18

Dura5on: 15 days (minimum)

Data of 2 par5cipants was not considered because they never accepted/dismissed any suggested rule.

Accepted Rules

80%79%

77%

75%

73%73%

67%

64%

57%50%

50%

47%

45%

34%

25%

20%

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16User

Rul

es C

ount

AcceptedRejected

Overall, 102 out of suggested 179 rules were accepted. 56.98% acceptance rate.

Accepted Rules

80%79%

77%

75%

73%73%

67%

64%

57%50%

50%

47%

45%

34%

25%

20%

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16User

Rul

es C

ount

AcceptedRejected

Overall, 102 out of suggested 179 rules were accepted. 56.98% acceptance rate.

70% (11 out of 16) users accepted 50% (and above) rules.

Filtered No5fica5ons

Average filter recall for 16 users: 45.81%

0

20

40

60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16User

Filte

r Rec

all (

%)

Filter Recall N-‐Auto N-‐Auto + N-‐Manual * = 100

N-‐Auto: Number of filtered no5fica5ons by the system N-‐Manual: Number of no5fica5ons dismissed manual

PrefMiner: Exit Ques5onnaire

Triggered aUer 15 days

Ques8on Average Response

I found the app useful for learning my preferences to filter no5fica5ons.

4.25

The app filtered most of the no5fica5ons that I didn’t want to receive.

4.33

The app incorrectly filtered no5fica5ons that I wanted to receive.

1.58

1:Strongly disagree -‐ 5:Strongly agree

•  First in-‐the-‐wild study for automa5c extrac5on of rules that reflect user’s preferences.

•  An open-‐source library (MyPref) that can learn the type of no5fica5ons that users would prefer to receive on their mobile phones in specific contexts.

•  PrefMiner represents a very effec5ve, yet transparent, solu5on for interrup5bility management for mobile devices.

Take Aways

42

Ques5ons?

Abhinav Mehrotra

Email: [email protected]

GitHub: hbps://github.com/AbhinavMehrotra

Play Store: hbps://play.google.com/store/apps/details?id=com.nsds.prefminer

prefminer:)mining)user’s)preferences)for ...axm514/paper/slides/ubicomp16.pdf ·...

Documents