work like a bee - taking advantage of diligent crowdsourcing workers at mediaeval 2014 crowdsourcing...
DESCRIPTION
Our presentation for the MediaEval 2014 Crowdsourcing Workshop. We showed that looking at the amount of HITs per worker can be a interesting thing to do.TRANSCRIPT
Work Like a BeeTaking Advantage of Diligent Crowdsourcing Workers
Presentation: Michael Riegler, Simula Research Laboratory, Norway Preben N. Oslen, Simula Research Laboratory Pål Halvorsen, Simula Research Laboratory [email protected]
The Idea• Use “good” workers
• Like the task or good with it
• Will do it much better
• So to speak diligent workers
• Do most of the work
• Extra: additional crowd workers
The Runs
• 4 different runs
• Run 1: MJV with additional crowdsourcing
• Run 2: Diligent Workers vote only
• Run3: MJV with weighted diligent workers
• Run4: Run3 with additional crowdsourcing
The ResultsRun WF1 score True Labels
Predicted Labels
R1 0.7207 183, 63, 291 192, 68, 279
R2 0.6919 183, 63, 291 208, 95, 234
R3 0.6912 183, 63, 291 208, 87, 242
R4 0.6919 183, 63, 291 208, 95, 234
Baseline 0.3809 183, 63, 291 0, 0, 537
The Analysis (i)
• Every run is better than the baseline
• R1 is the best performing run
• Small difference between R1 and R2 shows that diligent workers perform very good
WF1
sco
re0,68
0,693
0,705
0,718
0,73R1 R2 R3 R4
The Analysis (ii)
• This can for example be used to save costs
• R2, R3 and R4 perform similar
• Additional Crowdsourcing does not improve results
WF1
sco
re0,68
0,693
0,705
0,718
0,73R1 R2 R3 R4
The Conclusion• Very simple approach
• Showed that it can work
• Quality of workers is more important than the total number of workers
• No gain trough additional crowdsourcing
• Extend work on larger datasets
Tusen takk og ha det bra!
The End