The Scientific Community Game asA Crowdsourcing Platform to Distinguish Good from Bad

Presentation to Clientsby

Software Development Organization

4/24/2011 1

Crowdsourcing Platform

• Crowdsourcing – is the act of taking a job traditionally performed

by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.

– enlists a crowd of humans to help solve a problem defined by the system owners.

• A crowdsourcing platform is a generic tool that makes it easy to develop a crowdsourcing system.

4/24/2011 2

Crowdsourcing Platform

• The job, target problem is – to solve instances of a problem and make claims

about the solution process.– to build knowledge base of claims and techniques

to defend the claims

4/24/2011 3

Requirements for Crowdsourcing Platform

• Find a good way to combine user contributions to solve the target problem.

• Find a good way to evaluate users and their contributions.

• Find a good way to recruit and retain users.

4/24/2011 4

Combine user contributions

• We found an excellent way.• Users build on each others work:

strengthening.• Users check each others claims for correct

judgment.– Claims are defended and refuted.

• Users trade reputation and information.

4/24/2011 5

Voting with Justification

• I vote – for this claim (agree) because I can defend it and

refute its negation.– against this claim because I can oppose it (refute

or strengthen).

4/24/2011 6

Evaluate users and their contributions

• Calculate reputation– confidence by the proposer that a claim is good

(gc)– confidence by the opposer (refute or strengthen)

that the claim is bad (bc)

• The scholars are encouraged to set their confidences truthfully. Otherwise they don't gain enough reputation or they lose too much reputation.

4/24/2011 7

Reputation UpdateClaim good bad

propose up down

oppose down up

up: if you are good, there is a chance that you windown: if the other is good, there is a chance that you lose

up: reputation goes up, but has to provide knowledge that might reveal secret technique.down: reputation goes down, but might gain knowledge that reveals secret technique.

4/24/2011 8

Reputation UpdateClaim good bad

propose up down

oppose down up

up: if you are good, there is a chance that you windown: if the other is good, there is a chance that you lose

confidence: proposer: claim is good: gcopposer: claim is bad: bc

r = result of reputation protocol.Reputation update: r*gc*bc (various refinements are possible)

4/24/2011 9

Perfect

• Being perfect means to make perfect decisions.

• up: if you are perfect, you will not lose.• down: if the other is perfect, you will not win.

Claim good bad

propose up down

oppose down up

up: if you are good, there is a chance that you windown: if the other is good, there is a chance that you lose

4/24/2011 10

Result: We have successfully developed the SCG Crowdsourcing

platform: SCG Court• Requirements Document• User’s Guide• Continuous Testing Approach• Well documented software with Design

Document for Maintainers• Acceptance Test– CSP tournament– HSR tournament– NetworkFlow tournament

4/24/2011 11

Software Development Process

• Customized Spiral Process using continuous testing– http://groups.csail.mit.edu/pag/continuoustesting/

• Software Management Approach– Motivating software developers with interesting

project that provides numerous learning opportunities.

– Overcoming initial resistance by persistence and significant involvement of teaching staff. Adding Ahmed.

4/24/2011 12

Software Management Approach

• Hire good software developers with significant initiative and involvement in the project.

• Flexible project organization: very open to numerous requirements changes.

• Trusting software developers with significant responsibilities.

• Greek Gods: found subproject that was optimal for their time table.

• Dan Marcucci: Let’s use DemeterF the right way.

4/24/2011 13

Software Management Approach

• I amplified the good ideas that I spotted and did not force them on you!

• I continued to refine the game with Ahmed and Yue and we carefully justified all requirements changes.

• Reto: continuous testing approach. Setting up svn on assembla.

4/24/2011 14

Domain of Requests and Responses

• Three kinds of requests– Instance Requests

• Response: Instance• Boolean valid(Instance)

– Solution Requests• Response: Solution• Boolean valid(Instance,Solution)

– Claim Analysis Requests• Response: good/bad decision• determine which claims are

– good: agree– bad: refute or strengthen

• Make prediction about how protocol will run– Claims are about the relationship between instances and their

solutions

4/24/2011 15

What is good/bad?

• A claim is – good• if both scholars agree• if it is predominantly defended

– bad• if scholars don’t agree• if it is predominantly refuted

• Refutation is the complement of defense and is based on the requests and responses exchanged.

4/24/2011 16

Good/ Bad Meaningfor Mathematical Claims

• Good– true claim: if there is no strengthening relation

between claims (strengthen(c1,c2) is false if c1!=c2.)

– optimal claim: if there is a strengthening relation between claims.

• Bad– false claim– non-optimal claim

4/24/2011 17