University Of

HaifaIsrael

‘‘Friends Group’ Friends Group’ inin

Recommender SystemsRecommender Systems

Dr. Yuval Dan-Gur

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

2

Recommender SystemsThey are already hereThey are already here

IMDB

Ski-Europe Matchmaker

Amazon

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

3

Black boxesBlack boxes no transparency (Herlocker et al., 2000).

Exploration/Exploitation tradeoffExploration/Exploitation tradeoffrange of recommendations Vs. match level (Balabanovic, 1998).

Data sparseness and 'first rater‘Data sparseness and 'first rater‘number of raters compared to the number of items (Terveen and Hill, 2001).

Human tasteHuman taste non-linear and non-stable (Freedman, 1998; Pescovitz, 2000).

Recommender SystemsHow well do operational systems do?How well do operational systems do?

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

4

Recommendation giving and taking are social, biased processes - also in an automated system.

We reviewed possible influence and biases of several social and behavioral phenomena:

Social Comparison Theory (Festinger, 1954).

Attributing human qualities to computers (Nass and Moon, 2000).

Self-Serving Hypothesis in HCI (Nass and Moon, 1998).

Accepting advice from a system (Dijkstra, 1998, 1999; Dijkstra, Liebrand and Timminga, 1998; Murphy and Yetmar, 1996).

Diffusion of responsibility (Darley and Latane, 1968).

ELM - Elaboration Likelihood Model (Petty and Cacioppo, 1986).

Subjective value of information (Rafaeli and Raban, 2003).

Motivation for this researchMotivation for this research

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

5

Typical social aspects do not have an equivalent in recommender systems:

The ability to choose recommendation providers.

Querying for explanations is impossible in most systems (Herlocker et al., 2000; Preece, 1999; Terveen and Hill, 2001).

Filtering recommendation producers based on the item under concern or the situated environment.

Is recommendation a social process?Is recommendation a social process?

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

6

Human-computer dyad presents social rules and follows some established behavioral patterns (Moon

and Nass, 1998; Nass and Moon, 2000; Wood and Taylor, 1991).

We suggest that recommendation seeking is a natural social process that existed since the early days of tribal humanity (Cosley et al., 2003; Jungermann,

1999), and should be examined accordingly even when the process is automated.

ShouldShould automated recommendation procedure automated recommendation procedure be addressed as a social process? be addressed as a social process?

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

7

‘Friends group' describes a sub-group of ‘neighbors group’ that their characteristics are purposefully selected.

‘Friends group' differs from 'neighbors group‘:

The user is involved in forming the recommending group rather than relying upon an automatic procedure.

The user can choose the characteristics required for a recommendation provider to be included in the 'friends group'.

Number of recommenders may only decrease, leading to recommendations that may be OBEJECTIVEY – poorer.

The concept emerged mainly from the "Social Comparison Theory".

‘‘Friends Group’?Friends Group’?

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

8

1. Will users prefer to assume wider control over the recommendation process or to accept it as a "computerized oracle"?

2. Does the attitude of the recommendation seeker towards an advising group obey social rules (specifically the "social comparison" process), even when the user is aware that recommendations are processed and generated by a computerized system?

3. What are the characteristics of the 'friends' selected by the recommendation seeker to participate in his/her advising group when given an option to choose?

Research QuestionsResearch Questions

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

9

1.1. H1: Recommendation seekers will prefer to use H1: Recommendation seekers will prefer to use controlled 'friends groups' over automatically, controlled 'friends groups' over automatically, machine-generated 'neighbors groups'.machine-generated 'neighbors groups'.

2.2. H2: Recommendations produced by user-H2: Recommendations produced by user-controlled 'friends groups' will be more accepted controlled 'friends groups' will be more accepted and complied with by recommendation seekers and complied with by recommendation seekers than those produced by 'neighbors groups'.than those produced by 'neighbors groups'.

3.3. H3: Recommendation seekers will choose H3: Recommendation seekers will choose personally-similar 'friends' for their advising personally-similar 'friends' for their advising group.group.

Research HypothesesResearch Hypotheses

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

10

System available at http://qsia.org.

QSIA is a collaborative system for collection, management, sharing and assignment of knowledge items for learning. The system consists of various modules that allow the creation and editing of learning items, conducting online educational tasks and recommendation module that assists users in filtering relevant information.

QSIA supports both user-controlled ‘Friends’ advising group and auto-formed ‘Neighbors’ group.

Research Tool - QSIAResearch Tool - QSIA

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

11

User's involvement in the formation of the advising group.

Immediate usage of the "liked" recommended items in the same system.

Applying recommender technology to knowledge items for distance learning - not "natural" for recommender systems.

Research Tool - QSIA Research Tool - QSIA - continued

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

12

User has to choose

Recommendation seeking Self Browsing

Friends

User chooses characteristics of Friends

Recommendation list F

Acceptance of items

Acceptance/Rejection

Neighbors

Recommendation list N

Acceptance of items

Acceptance/Rejection

Interacting with QSIA - Conceptual Model

H1

H3

H2

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

13

‘Friends Group’ Characteristics in QSIA

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

14

Period of the field study - 2002 to 2004.

Number of users (teachers and students) –approximately 3000, most of them students.

Number of items - 10,000.

Served item-requests - 31,000. Mainly by self-browsing.

Item rankings – 3000, evaluated by around 300 users.

Study groups – 183.

Recommendations seeking data (either friends or neighbors), includes 895 requests (818 by students and 77 by teachers) generated by 108 active users.

Research - DataResearch - Data

2007ChaisYuval Dan-Gur15

Recom.Seeker

Characteristics of recommendation seeker in QSIA:

•Group membership.•Grade level.•Status/role.

Decision on Source of Recom.: Friends or Neighbors.

SoR=Fg

SoR=Ng

User choice of Friends'

characteristics.

Recom. engine with friends' algorithm.

Recom. engine with neighbors' algorithm.

Recom.Seeker

Recom. list when SoR= Ng

Recom. list when SoR= Fg

Recom. list when SoR= Fg

Recom. list when SoR= Ng Accepted

Accepted

Rejected

Rejected

Items used (in various forms and actions) by the user in QSIA system.

Whole population of users, their profiles, and a database of users' evaluations of

items.H1 is

examined here

H3 is examined

here

H2 is examined

here

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

16

Results - 1Results - 1The "Depth of Use" (DoUj), a variable that represents the maximum number of times that the jth user had asked for recommendations.

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

17

Results - 2Results - 2Instances that consist of three or more data scores (DoU≥3)

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

18

Results - 3Results - 3Acceptance of a recommended item is counted whenever the recommendation seeker "applies the item" or "simulates an item".Rejection of a recommended item is counted whenever the recommendation seeker restricts the consecutive action to only "viewing the item".

Table 6. Acceptance Ratio – Neighbors Table 7. Acceptance Ratio – Friends

Total acceptance

j

jiA

Total rejection

j

jiR

Total acceptance

j

jiA

Total rejection

j

jiR

Value 305 436 Value 174 128

Grand total 741 (by 38 users) Grand total 302 (by 32 users)

Std. Dev. 12.6 27.6 Std. Dev. 13.8 13.3

Acceptance ratio

41% Acceptance

ratio 58%

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

19

Results - 4Results - 4Acceptances and rejections for the same userssame users who asked for recommendations from both sourcesboth sources

Acceptance Ratios According to SoR

SoR=FgSoR=Ng

Number of records264377

Number of users19

Std. Dev.0.290.3

Mean acceptance ratio70%56%

Mean difference14%

α (Wilcoxon, one tailed)0.050

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

20

Results - 5Results - 5Users who asked for recommendations only from one one sourcesource, either 'friends groups' or 'neighbors groups'

Table 9. Acceptance Ratios of Exclusive Users

SoR=Fg SoR=Ng

Number of records 38 364

Number of users 13 19

Std. Dev. 0.38 0.33

Mean of acceptance ratio 70% 46%

Mean difference 24%

α (Binomial, one tailed) 0.037

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

21

Results - 5 Results - 5 - continuedWe identified We identified 36 items36 items that have the following characteristics: that have the following characteristics:These items were These items were recommended to users by bothrecommended to users by both 'friends groups' and 'neighbors groups'. 'friends groups' and 'neighbors groups'.Users acted upon theseUsers acted upon these recommendations, either by rejection or by acceptance in both scenarios of recommendations, either by rejection or by acceptance in both scenarios of SoR's.SoR's.Our intention is to test the difference in acceptance and rejection ratios of the same items, when they Our intention is to test the difference in acceptance and rejection ratios of the same items, when they were offered to users by 'friends groups' and 'neighbors groups', were offered to users by 'friends groups' and 'neighbors groups', based on 394 usage records.based on 394 usage records.

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

22

Results - 5 Results - 5 - continued

Table 10. Users' Acceptance Ratios for Items that were Offered by Both Sources

Acceptance ratio

Rejection ratio

Total

SoR=Fg 142 (50.9%)

137 (49.1%)

279 (100%)

SoR=Ng 66 (57.4%)

49 (42.6%)

115 (100%)

Total 208 (52.8%)

186 (47.2%)

394 (100%)

Ratio difference 6.5%

α (Pearson Chi-Square with continuity correction) 0.28

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

23

Results - 6Results - 6FRI (Frequently Recommended Items) = Items recommended approximately 5 times more frequent than the average recommendations number of an item.

Table 11. Acceptance Ratios of the Most Frequently Recommended Items

SoR=Fg SoR=Ng

Number of recommendations 93

Number of items 4

Mean of acceptance ratio 63.9% 48.7%

Std. Dev. 0.033 0.056

Mean difference 15.2%

α (Wilcoxon, one tailed) 0.034

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

24

Results - 7Results - 7Friends characteristic = Group

Table 13. Group Choice Proportions

Similar Dissimilar

Proportion 88.3%

(144 records) 11.7%

(19 records)

Difference 76.6%

Significance level (Sign test) α<0.0001

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

25

Results - 8Results - 8Friends characteristic = Role

Table 14. Status/Role Choice by Teachers and Students

User's choice

Teacher Student Total

Teacher 5.8% (5)

0% (0)

5.8% (5)

Student 68.6% (59)

25.6% (22)

94.2% (81)

Use

r's

role

Total 74.4% (64)

25.6% (22)

100% (86)

Difference in students' choices

43% α (sign test) < 0.0001

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

26

First HypothesisFirst Hypothesis: Recommendation seekers will prefer to : Recommendation seekers will prefer to use controlled 'friends groups' over automatically, use controlled 'friends groups' over automatically, machine-generated 'neighbors groups‘machine-generated 'neighbors groups‘

Supported: Users do develop a tendency to choose 'friends group' recommendations. The probability of this tendency increases in as more recommendations are sought.

Also, "experienced" users choose 'friends groups' significantly more than "new" users.

Main Findings -First HypothesisMain Findings -First Hypothesis

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

27

Second HypothesisSecond Hypothesis: Recommendations produced : Recommendations produced by user-controlled 'friends groups' will be more by user-controlled 'friends groups' will be more accepted and complied with by recommendation accepted and complied with by recommendation seekers than those produced by 'neighbors seekers than those produced by 'neighbors groups‘.groups‘.

Supported: We found a positive significant difference in the mean ratio of acceptance when we tested all users who had received and acted upon recommendations from both sources ('friends group' and 'neighbors group').

Main Findings -Second HypothesisMain Findings -Second Hypothesis

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

28

There was a higher positive significant difference in the mean acceptance ratios (24%, α = 0.037) for users who received recommendations from only one source (either 'friends group' or 'neighbors group').

Also, when the same items were offered to users from both sources (N=36), the acceptance level was 6.5% higher when the recommendations were offered by 'friends groups' (P-value= 0.28).

For the most frequently recommended items that were recommended by both 'friends group' and 'neighbors group', the acceptance ratio was 15.2% higher (N=4, α = 0.034) for the same items when they were recommended by 'friends groups'.

Main Findings -Second Hypothesis Main Findings -Second Hypothesis - continued

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

29

Third Hypothesis: Recommendation seekers will Third Hypothesis: Recommendation seekers will choose personally-similar 'friends' for their advising choose personally-similar 'friends' for their advising group.group.

Partially supported: There were many missing values in this part of our dataset: in almost half the records users made a group choice, in another quarter of the cases they made a role choice, and in only approximately 6% of the cases did users make a grade choice.

Main Findings -Third HypothesisMain Findings -Third Hypothesis

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

30

We analyzed the characteristics independently and found that in accordance with our hypothesis, users significantly prefer their own group over other groups (76.6%, α<0.0001).

Role: students asked for teachers' recommendations 43% more than for students' recommendations (α<0.0001). We explained it by an alternative hypothesis stating that students choose teachers' recommendations because of their role authority and knowledge expertise (Wyeth and Watson, 1971).

Main Findings -Third Hypothesis Main Findings -Third Hypothesis - continued

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

31

Over time, users increasingly sought recommendations users increasingly sought recommendations from 'friends groups'from 'friends groups' and the probability to do so increased with higher use of recommendations.

Users' acceptance level of recommendations was higher acceptance level of recommendations was higher when they asked for 'friends groups' recommendationswhen they asked for 'friends groups' recommendations. In addition, the same items were more readily accepted when offered to the user by the 'friends group' than when offered by the 'neighbors group'. The difference in acceptance was higher for items that were recommended frequently.

We had insufficient data for some of the planned statistical tests of the third hypothesis. Nevertheless, we concluded that own group choice was the most important characteristic own group choice was the most important characteristic for usersfor users to assign to their advising group members.

So, what is new?So, what is new?

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

32

Absence of a comparable field study.

We did not collect self-report of user motivations for their actions - we only collected data on the dependent variables and deduced the users' behavior.

The participating populations, except in one case, were homogeneous: students and teachers of academic institutions.

The characteristics of the advising group that were possible for the recommendation seeker to control were limited: group, grade level and role.

QSIA - we did not trace similar systems as a benchmark for its unique characteristics.

Lack of data - the algorithm of 'friends group' lowers the number of members of the "advising group" because of the characteristics' constraints, and thus, may produce recommendations of less objective quality.

Weaknesses and Limitations Weaknesses and Limitations

2007ChaisYuval Dan-Gur

University Of

HaifaIsrael

33

The research demonstrates that acceptance likelihoodacceptance likelihood among users of social collaborative systems of the recommendations depends on the groupgroup that made the recommendations and on the users' involvementusers' involvement in the formation of that group.

The main new aspect of our findings is the relationship between the perceived, subjective qualityperceived, subjective quality of the recommendation, and the user's involvement in the formation of the advising group.

SummarySummary

University Of

HaifaIsrael

Thank you!Questions?