wisdom of crowds and rank aggregation
DESCRIPTION
Wisdom of Crowds and Rank Aggregation. Mark Steyvers Department of Cognitive Sciences University of California, Irvine. Joint work with: Brent Miller, Pernille Hemmer, Mike Yi, Michael Lee. Wisdom of crowds phenomenon. - PowerPoint PPT PresentationTRANSCRIPT
Wisdom of Crowds and Rank Aggregation
Mark SteyversDepartment of Cognitive Sciences
University of California, Irvine
Joint work with:Brent Miller, Pernille Hemmer, Mike Yi, Michael Lee
Wisdom of crowds phenomenon
Aggregating over individuals in a group often leads to an estimate that is better than any of the individual estimates
2
Examples of wisdom of crowds phenomenon
3
Galton’s Ox (1907): Median of individual weight estimates came close to true answer
Prediction markets
Ulysses S. Grant
James Garfield
Rutherford B. Hayes
Abraham Lincoln
Andrew Johnson
James Garfield
Ulysses S. Grant
Rutherford B. Hayes
Andrew Johnson
Abraham Lincoln
Our research: ranking problems
time
What is the correct chronological order?
Aggregating ranking data
5
D A B C A B D C B A D C A C B D A D B C
Aggregation Algorithm
A B C D A B C D
ground truth
=?
group answer
Task constraints
No communication between individuals
There is always a true answer (ground truth)
Unsupervised algorithms no feedback is available ground truth only used for evaluation
6
Classic models: Thurstone (1927) Mallows (1957); Fligner and Verducci, 1986 Diaconis (1989) Voting methods: e.g. Borda count (1770)
Machine learning applications Information retrieval and meta-search
e.g. Klementiev, Roth et al. (2008; 2009), Lebanon & Mao (2008); Dwork et al. (2001)
multi-object tracking e.g. Huan, Guestrin, Guibas (2009); Kondor, Howard, Jebara (2007)
Unsupervised models for ranking data
7Many models were developed for preference rankings and voting situations no known ground truth
Unsupervised Approach
8
D A B C A B D C B A D C A C B D A D B C
Generative Model
? ? ? ?
latent ground truth
Incorporate individual differences
Overview of talk
Reconstruct the order of US presidents
Effect of group size and expertise
Reconstruct the order of events
Traveling Salesman Problem
9
Experiment: 26 individuals order all 44 US presidents
10
George Washington John Adams Thomas Jefferson James Madison
James Monroe John Quincy Adams Andrew Jackson Martin Van Buren
William Henry Harrison John Tyler James Knox Polk Zachary Taylor
Millard Fillmore Franklin Pierce James Buchanan Abraham Lincoln
Andrew Johnson Ulysses S. Grant Rutherford B. Hayes James Garfield
Chester Arthur Grover Cleveland 1 Benjamin Harrison Grover Cleveland 2
William McKinley Theodore Roosevelt William Howard Taft Woodrow Wilson
Warren Harding Calvin Coolidge Herbert Hoover Franklin D. Roosevelt
Harry S. Truman Dwight Eisenhower John F. Kennedy Lyndon B. Johnson
Richard Nixon Gerald Ford James Carter Ronald Reagan
George H.W. Bush William Clinton George W. Bush Barack Obama
= 1= 1+1Measuring performance
Kendall’s Tau: The number of adjacent pair-wise swaps
Ordering by IndividualA B E C D
True OrderA B C D E
C DEA B
A B E C D
A B C D E= 2
Empirical Results
12
1 10 200
100
200
300
400
500
Individuals (ordered from best to worst)
(random guessing)
Thurstonian Model
13
A. George Washington
B. James Madison
C. Andrew Jackson
Each item has a true coordinate on some dimension
Thurstonian Model
14
… but there is noise because of encoding errors
A. George Washington
B. James Madison
C. Andrew Jackson
Thurstonian Model
15
A. George Washington
B. James Madison
C. Andrew Jackson
Each person’s mental encoding is based on a single sample from each distribution
A
B
C
Thurstonian Model
16
A. George Washington
B. James Madison
C. Andrew Jackson
A
B
C
A < C < B
The observed ordering is based on the ordering of the samples
Thurstonian Model
17
A. George Washington
B. James Madison
C. Andrew Jackson
A
B
C
A < B < C
The observed ordering is based on the ordering of the samples
Thurstonian Model
18
A. George Washington
B. James Madison
C. Andrew Jackson
Important assumption: across individuals, standard deviation can vary but not the means
Graphical Model of Extended Thurstonian Model
19
j individuals
jx
jy
μ
j
| , ~ N ,ij j jx
( )j jranky x
~ Gamma ,1 /j
Latent group means
Individual noise level
Mental representation
Observed ordering
Inferred Distributions for 44 US Presidents
20
George Washington (1)John Adams (2)
Thomas Jefferson (3)James Madison (4)James Monroe (6)
John Quincy Adams (5)Andrew Jackson (7)
Martin Van Buren (8)William Henry Harrison (21)
John Tyler (10)James Knox Polk (18)
Zachary Taylor (16)Millard Fillmore (11)Franklin Pierce (19)
James Buchanan (13)Abraham Lincoln (9)
Andrew Johnson (12)Ulysses S. Grant (17)
Rutherford B. Hayes (20)James Garfield (22)Chester Arthur (15)
Grover Cleveland 1 (23)Benjamin Harrison (14)
Grover Cleveland 2 (25)William McKinley (24)
Theodore Roosevelt (29)William Howard Taft (27)
Woodrow Wilson (30)Warren Harding (26)Calvin Coolidge (28)Herbert Hoover (31)
Franklin D. Roosevelt (32)Harry S. Truman (33)
Dwight Eisenhower (34)John F. Kennedy (37)
Lyndon B. Johnson (36)Richard Nixon (39)
Gerald Ford (35)James Carter (38)
Ronald Reagan (40)George H.W. Bush (41)
William Clinton (42)George W. Bush (43)
Barack Obama (44)
error bars = median and minimum sigma
Calibration of individuals
21
0 0.1 0.2 0.3 0.450
100
150
200
250
300
R=0.941
inferred noise level for
each individual
distance to ground
truth
individual
Wisdom of crowds effect
22
1 10 200
50
100
150
200
250
300
350
Individuals
Thurstonian ModelPerturbationIndividuals
Alternative Heuristic Models
Many heuristic methods from voting theory E.g., Borda count method
Suppose we have 10 items assign a count of 10 to first item, 9 for second item, etc add counts over individuals order items by the Borda count
i.e., rank by average rank across people
23
Model Comparison
24
1 10 20 300
50
100
150
200
250
300
350
Individuals
Thurstonian ModelPerturbationBorda countIndividuals
Borda
Overview of talk
Reconstruct the order of US presidents
Effect of group size and expertise
Reconstruct the order of events
Traveling Salesman Problem
25
Experiment
78 participants 17 ordering problems each with 10 items
Chronological Events Physical Measures Purely ordinal problems, e.g.
Ten Amendments Ten commandments
26
Ordering states west-east
27
Oregon (1)
Utah (2)
Nebraska (3)
Iowa (4)
Alabama (6)
Ohio (5)
Virginia (7)
Delaware (8)
Connecticut (9)
Maine (10)
Ordering Ten Amendments
28
Freedom of speech & religion (1)
Right to bear arms (2)
No quartering of soldiers (4)
No unreasonable searches (3)
Due process (5)
Trial by Jury (6)
Civil Trial by Jury (7)
No cruel punishment (8)
Right to non-specified rights (10)
Power for the States & People (9)
Ordering Ten Commandments
29
Worship any other God (1)
Make a graven image (7)
Take the Lord's name in vain (2)
Break the Sabbath (3)
Dishonor your parents (4)
Murder (6)
Commit adultery (8)
Steal (5)
Bear false witness (9)
Covet (10)
Effect of Group Size: random subgroups
30
0 10 20 30 40 50 60 70 807
8
9
10
11
12
13
14
Group Size
T=0T=2
T=12
How effective are small groups of experts?
Want to find experts endogenously – without feedback
Approach: select individuals with the smallest estimated noise levels based on previous tasks
We are identifying general expertise (“Pearson’s g”)
31
Group Composition based on prior performance
32
0 10 20 30 40 50 60 70 807
8
9
10
11
12
13
14
Group Size
T=0T=2
T=12
T = 0
# previous tasks
T = 2T = 8
Group size (best individuals first)
33
Endogenous no feedback
required
Exogenous selecting people based on
actual performance
0 10 20 30 407
8
9
10
11
12
13
14
0 20 407
8
9
10
11
12
13
14
Overview of talk
Reconstruct the order of US presidents
Effect of group size and expertise
Reconstruct the order of events
Traveling Salesman Problem
34
Recollecting Order from Episodic Memory
35
Study this sequence of images
Place the images in correct sequence (serial recall)
36
A
B
C
D
E
F
G
H
I
J
Average results across 6 problems
37
Mea
n
1 10 20 300
5
10
15
Individuals
Thurstonian ModelPerturbation ModelBorda countIndividuals
Calibration of individuals
38
0 2 4 60
5
10
15
20
25
30
R=0.920
inferred noise level
distance to ground
truth
individual
(pizza sequence; perturbation model)
Overview of talk
Reconstruct the order of US presidents
Effect of group size and expertise
Reconstruct the order of events
Traveling Salesman Problem
39
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21Find the shortest route between cities
40
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21 - subj 5
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21 - subj 83
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21 - subj 60
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21
B30-21
Individual 5 Individual 83 Individual 60Optimal
Dataset Vickers, Bovet, Lee, & Hughes (2003)
83 participants 7 problems of 30 cities
TSP Aggregation Problem
Data consists of city order only No access to city locations
42
Heuristic Approach
Idea: find tours with edges for which many individuals agree
Calculate agreement matrix A A = n × n matrix, where n is the number of cities aij indicates the number of participants that connect cities i and j.
Find tour that maximizes
43
tourji
cija
),(
(this itself is a non-Euclidian TSP problem)
Line thickness = agreement
44
1
2
3
45
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2021
22
23
24
25
26
27
28
29
30
B30-21Blue = Aggregate Tour
45
Results averaged across 7 problems
0
2
4
6
8
10
12
14
16
18
Per
cent
ove
r Opt
imal
aggregate
Summary Combine ordering / ranking data
going beyond numerical estimates or multiple choice questions
Incorporate individual differences assume some individuals might be “experts” going beyond models that treat every vote equally
Applications combine multiple eyewitness accounts combine solutions in complex problem-solving situations fantasy football
47
Predictive Rankings: fantasy football
49
South Australian Football League (32 people rank 9 teams)
1 10 20 300
20
40
60
80
Individuals
Thurstonian ModelPerturbation ModelBorda countIndividuals
Australian Football League (29 people rank 16 teams)
1 10 20 300
5
10
15
20
25
Individuals
1 10 20 300
20
40
60
80
0.8 1 1.2 1.4 1.6 1.8
0
2
4
6
8
10
12
14
16
18R=-0.752
1
2
3
4
5
6
7
8
9
10
1112
13
14
15
16
17
Predicting problem difficulty
50
std
dispersion of noise levels across individual
distance of group
answer to ground truth
ordering states geographically
city size rankings
Related Concepts in Supervised Learning
Boosting combining multiple classifiers
Bagging (Bootstrap Aggregating)
51