the future of crowd work aniket kittur et al., (2013). the future of crowd work , cscw’13
Post on 23-Feb-2016
95 Views
Preview:
DESCRIPTION
TRANSCRIPT
The future of crowd workAniket Kittur et al., (2013). The future of crowd work, CSCW’13
Mejdl SafranFall 2013
SIUC
More complex, creative and highly valued work
“a platform is needed for managing pools of tasks and workers. Complex tasks must be decomposed into smaller subtasks, each designed with particular needs and characteristics which must be assigned to appropriate groups of workers who themselves must be properly motivated, selected (e.g., through reputation), and organized (e.g., through hierarchy). Tasks may be structured through multi-stage workflows in which workers may collaborate either synchronously or asynchronously. As part of this, AI may guide (and be guided by) crow workers. Finally, quality assurance is needed to ensure each worker’s output is of high quality and fits together.”
Research challenges in 12 major areas
1. The future of crowd work processes2. The future of crowd computation3. The future of crowd workers
Crowd Work: Workflow
Motivation/Goals Complex crowd work cannot accomplished by simple parallel
approaches (e.g., aggregating multiple independent judgments) Workflows are needed that facilitate decomposing tasks into
subtasks, managing the dependencies between subtasks, and assembling the results.
Crowd Work: Workflow
Research Proposal Improving existing workflows toward more general tasks and complex
problems that have no clearly defined solution (creativity, civic planning, etc.)
Comparing to simple workflow, crowd workflow could involve a much larger scale of operation, an a much more heterogeneous set of actors
How to validate? We must experiment on a large space of parameters, incentives and
decompositions. Costs of doing so can be reduced through models of worker behavior,
or by reusing proven design patterns..
Crowd Work: Task Assignment
Motivation/Goals Sharing limited resources (workers) requires coordination. While workers are continuously employed with tasks that
match their interests, requesters will see their tasks completed quickly.
ESP game and Galaxy Zoo use FCFS model oDesk and AMT use market model
Crowd Work: Task Assignment
Research Proposal Data partitioning, OS scheduling and failover! Or new theoretical model. “Several workers in our survey complained that they spent too
much energy finding appropriate tasks” (Task Recommendation).
Research question: should tasks be pulled by workers or pushed by platforms?
Crowd Work: Quality Control
Motivation/Goals Why? High throughput, low transaction costs ands
complex/subjective tasks. Cheating or game the system. Workers with low expertise and requesters with unclear
instructions.
Crowd Work: Quality Control
Research Proposal Up-front task design: peer-review, agreement filters (writing an
essay!!), optimize instructions, etc. Post-hoc result analysis: gold standard data (prevent biases from
dominating the results) and workers’ votes (costs more). Better approach: use ML to predict the quality of a worker’s
output from their behavior. Open-ended work and highly skilled tasks?? Is it possible to robustly measure worker’s skills at tasks such as
audio engineering and poetry? Should we rely on peer evaluation?
Crowd Work: Real-time Crowd Work
Motivation/Goals We need to create flash crowds: groups of individuals who
arrive moments after a request and can work synchronously. All on-demand CS are limited by the crowd latency.
Crowd Work: Real-time Crowd Work
Research Proposal Fast recruitment is a major research area in real-time CS. Time-limited tasks (e.g., search for a missing person and timed
competitions) E-mail a set of workers the night before the study and
announced a time for the experiment. Paying workers a small wage to stay on call. They need to be modeled using queuing theory. Scaling up to increased demand for real-time workers Making workers efficient enough to collectively generate
results ahead of time deadlines
Crowd Computation: Crowds Guiding AIs
Motivation/Goals In human computation, people act as computational
components and perform the work that AI systems lack the skills to complete.
By tapping into crowd intelligence, computational systems can support a much broader set of tasks
Crowd workers may end up training machines to replace them!!
Gathered data is used to train algorithms (e.g., Google image search engine, birds count project)
Crowd Computation: Crowds Guiding AIs
Research Proposal Integrating crowds more deeply into algorithms. Design ML algorithms that more deeply understand the human
nature of these labels.
Crowd Computation: AIs Guiding Crowds
Motivation/Goals Humans can introduce errors, and errors are amplified as they
propagate through the crowd. Integrate crowds inside of software and use the software to
help guide crowd work. E.g., ML model can determine which work products may still
be improved and then assign workers most likely to make such improvements.
Crowd Computation: AIs Guiding Crowds
Research Proposal AIs can point out what others have done in similar contexts. We need to move from a setting where simple AIs completely
determine a workflow to a richer, mixed setting where crowds and AIs jointly teach each other, and jointly control the work process.
Crowd Computation: Crowdsourcing Platforms
Motivation/Goals Limiting ourselves to exiting platforms greatly restricts the
scope and nature of change we can enact.
Crowd Computation: Crowdsourcing Platforms
Research Proposal Different platforms for different aims. MobileWorks (mClerk): through SMS, more populations. Vending machine with purpose (exam grading) targeting
experts. Labor exchanged policy issues How can platforms disclose enough info to be trusted as a
source of worker quality while also maintaining privacy? Security concerns both attacks on platforms or use of
platforms for launching attacks
Crowd Workers: Reputation
Motivation/Goals “Graduated from UC Berkeley and working Google” Traditional employers judge a prospective employee's
education and work history through interviews, transcripts, and references.
“I have an accepting rate of 4 out of 5 on AMT”
Crowd Workers: Reputation
Research Proposal We want better reputation rankings for workers to be
established with the platforms Reputation systems must be robust to cheating and gaming
while supporting low-transaction cost hiring. One way: a web of trust! (workers and requesters validate each
other. What if malicious build their own web of trust!!! (gaming the
system) Technical tools for sharing information about workers
requester abuses!! (privacy)
Crowd Workers: Motivation & Rewards
Motivation/Goals Crowd workers are a diverse and multifaceted population with
a range of motives and experiences Worker said “Mturk should encourage and reward requesters
that provide clear instructions, quick payment, and higher pay”
Crowd Workers: Motivation & Rewards
Research Proposal Past research:
Requesters should clearly understand and communicate desired behaviors
Understand and worker motivations and incentives with these desired behaviors
Design the requests and incentive structures in order to achieve both effective task completion and worker satisfaction
Beyond financial incentives Build systems that support workers’ diverse motivations.
top related