20120106 hicss 45-goers_collaborative-idea-selection
TRANSCRIPT
A New Collaborative Algorithm for the Selection of Ideas
by Jana Goers, Graham Horton and Nadine Kempe
Group of experts screening ideas
Challenges in Idea Selection
Situation
Many ideas
Ill-defined ideas
Group of experts
Problems
Lack of consensus of idea selection leads to a lack of commitment for implementation
High cognitive load when forced to select among 100 or more ideas
Rejecting good ideas by mistakeGroup Task
Select the ideas which areworth pursuing
Requirements:
1. Fast selection process
2. Consensus about selection result
3. Low cognitive load
No method that fulfills these requirements.
Requirements For a Good Idea Selection
This would deliver: High acceptance of result
But: High cognitive load
Slow
Alternative 1
We could use a group discussion.
This would deliver: Fast selection procedure
Low cognitive load
But: Low acceptance of result
Alternative 2
Or we could partition the task.
A method that combines:
Speed and cognitive load comparable to parallelised selection
Consensus comparable to a group discussion
But what could this collaborative approach look like?
Is There Something in Between?
"Divide and Conquer" principle:
The given problem is too big to solve.
1. So, we divide the problem...
2. ... and solve each sub-problem.
3. We obtain the overall solution by putting together the sub-solutions.
A Principle of Computer Science
Adaption of "Divide and Conquer" for idea selection:
1 Partition the ideas among the participants.
2a) Perform local selections on each partition.(Parallel work; establishes local idea quality)
3 Perform selection of partition representatives.(Group work; establishes global idea quality)
2b) Revise the local selections.(Parallel work; corrects local selections according to global quality)
4 The overall selection is now the unification of the local selections.
Sketch of Our Approach
Step 1: Partition pool of ideas into subsets of five
The Threshold Algorithm
Step 2: Assign each subset to a group member
The Threshold Algorithm
Step 3: Each group member performs local selection
The Threshold Algorithm
locally selected ideas
locally rejected ideas
Step 4: Each group member selects ideas for his subset
The Threshold Algorithm
Mark worst idea of your selected ideas! locally selected
ideas
locally rejected ideas
local threshold idea
Result (4): Five local selections based on local quality
The Threshold Algorithm
Step 5: Group discussion only of local threshold ideas
The Threshold Algorithm
Step 6: Group determines global threshold idea
The Threshold Algorithm
The Threshold Algorithm
The Threshold Algorithm
Step 7: Each group member revises local selection
What were the measurements? Actual and perceived duration
Cognitive load
Acceptance of results
Number of rejection errors
Experiments
Parallel Threshold Discussion
Group 1 Idea Set 1 Idea Set 2 Idea Set 3
Group 2 Idea Set 2 Idea Set 3 Idea Set 1
Group 3 Idea Set 3 Idea Set 1 Idea Set 2
Group 4 Idea Set 1 Idea Set 2 Idea Set 3
The threshold approach ... needs twice the time of the parallelised method,
but only half the time of the discussion.
Actual Duration in Minutes
12:06:00 AM
12:11:45 AM
12:23:40 AM
12:00:00 AM
12:07:12 AM
12:14:24 AM
12:21:36 AM
Du
rati
on
in M
inu
tes
Parallel Threshold Discussion
But More Interesting Is the Perceived Speed
Even though the threshold needs twice the time of the parallel method, ... it is on the same subjective speed level as the parallel method.
81 80
58
50
60
70
80
90
100
Sum
me
d P
oin
ts o
f Ev
alu
atio
n F
orm
Parallel Threshold Discussion
Cognitive Load
The cognitive load of the threshold approach ... is not as high as for the discussion but not as low as for the parallel
method.
30
39
46
0
10
20
30
40
50
Sum
me
d P
oin
ts o
f Ev
alu
atio
n F
orm
Parallel Threshold Discussion
Acceptance of Selection Results
Unfortunately, the threshold approach ... has a similar subjective acceptance as the parallel method.
7476
86
60
70
80
90
100
Sum
me
d P
oin
ts o
f Ev
alu
atio
n F
orm
Parallel Threshold Discussion
However, the threshold approach ... produces as few rejection errors as the discussion.
Number of Rejection Errors
33
1816
0
10
20
30
40
Nu
mb
er
of
reje
ctio
n e
rro
rs
Parallel Threshold Discussion
But why such a bad performance regarding acceptance?
Our suspicion: Participants mistrusted the algorithm due to lack of understanding
A Master Thesis is currently investigating this.
Acceptance and Rejection Errors
Our approach was able to fulfill our requirements:
1. Fast selection process
2. Consensus in selection result few rejection errors
3. Low cognitive load
The threshold method is an appropriate trade-off.
Conclusion
Our next steps:
Could a better understanding of the method improve acceptance?
Could we reduce cognitive load by using pairwise comparisons?
What could a multi-criteria approach look like?
Could we obtain more speed by using a computer-supported method?
Are there other applications?
Outlook
1. Individual and group tasks: We learned that a collaborative selection doesn't have to be
executed entirely by a group.
Are there other collaborative tasks which can benefit from the "Divide and Conquer" principle?
2. Abstraction of what we did: We designed a collaborative selection process by using a principle of
Computer Science.
Which other principles could be of value for collaborative tasks?
Two Insights