effects of expertise assessment on the quality of task routing in human computation

Copyright 2010 Digital Enterprise Research Institute. All rights reserved.

Digital Enterprise Research Institute www.deri.ie

EFFECTS OF EXPERTISE ASSESSMENT ON THE QUALITY OF TASK ROUTING

IN HUMAN COMPUTATION

Umair ul Hassan, Sean O’Riain, Edward CurryDigital Enterprise Research Institute

National University of Ireland, Galway

International Workshop on Social Media for Crowdsourcing and Human Computation - SoHuman’13, Paris, France


Agenda

Paper Overview Motivation

Human Computation Task Routing

Challenges of Push Routing Experiment

Use case Methodology Results

Summary

2


Paper Overview

Motivation People have differing levels of expertise Effective task routing requires expertise information Expertise profiling involves assessment

Problem How to assess worker’s expertise for generating profiles? How to reduce costs of expertise assessment while attaining

higher quality of task routing? Contribution

Comparison of self-assessment and task-assessment approaches

A hybrid approach, based on combination of self-assessment and task assessment, for cost reduction

3


Human Computation

Solve computationally hard problems with help of humans

Algorithms control human workers Computation is carried out by Humans

4

* Barowy et al, “AutoMan: a platform for integrating human-based and digital computation,” OOPSLA ’12

Define Compute

Algorithm

DeveloperWorkers


Human Computation

5

* Edith Law and Luis von Ahn, Human Computation - Core Research Questions and State of the Art

Input Output

Task Routerbefore computation

Output Aggregationafter computation

Task Designduring computation

Our Focus


Task Routing

Pull Routing System provides an interface to support workers Workers actively seek tasks and assign to themselves

6

Workers

Tasks Select

Result

Algorithm

Search & Browse Interface

* www.mtruk.com

Result


Task Routing

Push Routing System has complete control over assignment of tasks

– Based on criteria such as expertise, cost, and latency Workers passively receive tasks

7

Workers

Tasks

Assign

Result

Assign

Algorithm

Task Interface

* www.mobileworks.com

Result


Challenges of Push Routing

Workers have different domain knowledge and expertise

1. How to define the expertise requirements of a task? And how to model the expertise profile of a worker?

2. How to profile the expertise of human workers, via suitable expertise assessment methods with minimum cost?

3. How to leverage the expertise profiles of workers for effectively routing tasks , resulting in quality responses?

8


Routing

Profiling

Knowledge Profile

TasksPerformance

Profile

3. Test

Tasks

1. Concepts

Routing Model

5. New Tasks

2. Self

Assessment

4. Task

Assessment

6. Routed Tasks

Workers

Two phase process

Steps of push routing using worker profiles

9

Cost of assessment for profiling

Quality of profiles for routing


Use case: Verification Tasks

Data quality in DBpedia Verification of new facts for DBpedia

10

Concept related to the task



11

Update: Missing Value

dbpedia-owl:writer = dbpedia:Akiva_Goldsman

SKOS Concepts: American_biographical_filmsFilms_set_in_the_1950s

Worker Expertise SKOS Concepts:Films_set_in_the_1950s (Good)Films_about_psychiatry (Poor)American_drama_films (Fair)

Data Quality Algorithm

Workers & Expertise Model

Entity: A Beautiful Mind

SKOS Concepts:American_biographical_filmsFilms_set_in_the_1950s

Property & Values:dbpedia-owl:Work/runtime 135.0dbpedia-owl:director dbpedia:Ron_Howarddbpedia-owl:producer dbpedia:Ron_Howard dbpedia:Brian_Grazedbpedia-owl:starring dbpedia:Ed_Harris dbpedia:Russell_Crowe

Source Data

Task: Confirm Missing Value

Did Akiva Goldsman wrote the movie "A Beautiful Mind"?

SKOS Concepts:American_biographical_filmsFilms_set_in_the_1950s

Task Routing

MatchAmerican_biographical_filmsAmerican_drama_films (Fair)

Task Model

Routing Model* SKOS = Simple Knowledge Organization System



Datasets based on films related entities from hollywood and bollywood

Distribution of tasks against number of concepts per task

12

Dataset Characteristics

Movies

Dataset

Actors

Dataset

Total entities 724 14

Total concepts 42 14

Total tasks 230 120

Avg. tasks per concept 9 8.6

Avg. concepts per task 1.64 1

1 2 3 4 50

20

40

60

80

100

120

140

160

No. of Concepts per Task

No

. o

f T

ask

s


Profiling

To quantify expertise levels of workers against concepts is set of Workers, is set of Concepts, is set of Tasks

Worker profiles in the form of matrix is concept, is worker

13

Concept

c1: Buddy films 0.6 0.2 0.2

c2: Gang films 0.6 0.2 0.6

c3: Horror films 0.8 0.4 0.4

c4: Comedy films 0.8 0.6 0.6


Assessment

Self-assessment for generating knowledge profile Workers provide self-rating of knowledge for each concept Easier to implement but may not be accurate

14


Assessment

Task-assessment for generating performance profile Workers provide responses to assessment tasks Accurate in measuring expertise but difficult for diverse data

15


Routing

Leverage knowledge and performance profiles to match tasks with workers For each task and worker calculate ranking score based on concepts

associated with task in worker’s profiles Assign the task to top-1 worker in the ranked list

Ranking Methods RND: assign random value between 0 and 1

SA: average of values in worker’s knowledge profile

TA: average of values in worker’s performance profile

CA: average of element-wise multiplication of values in worker’s knowledge and performance profiles

16


Knowledge workers

Volunteers having varying knowledge about films Hollywood vs.

Bollywood

Survey before and after participation

17

Movies

Dataset

Actors

Dataset

No. of knowledge workers (volunteers) 11 26

No. of knowledge concepts 42 14

No. of test tasks (profiling phase) 100 56

No. of new tasks (routing phase) 130 64

Interest Knowledge Expertise Confidence0

1

2

3

4

5

6

7

8

9

10Before

After

Avera

ge L

evel

Only significant difference


Evaluation

Metrics Quality (for routed tasks during routing phase)

– Response Rate: percentage of routed tasks with agree or disagree responses

– Accuracy: percentage of routed tasks with correct responses

Cost (for assessments during profiling phase)– Workload: number of decisions for self-rating of

conceptual knowledge or responding to test task

Hypothesis The quality of CA strategy approaches the quality

of TA strategy during routing phase, while requiring comparatively less assessment cost during profiling phase.

18


Results: Costs

Combined assessment Filtering assessment tasks based on highly self-rated

concepts reduces assessment cost

19

RND SA TA CA CA (P+) CA (F+) CA (G+) CA (Ex)0%

20%

40%

60%

80%

100%

120%

140%

160%

Movies Dataset Actors Dataset

% w

okrl

oad c

om

pare

d t

o T

A

For examples filter tasks with concepts of Good or higher self-rating


Results: Quality of Routing

Likelihood of response and accuracy of response remains near maximum during routing stage

20

RND SA TA CA CA (P+)

CA (F+)

CA (G+)

CA (Ex)

0%

20%

40%

60%

80%

100%


% A

ccura

cy

RND SA TA CA CA (P+)

CA (F+)

CA (G+)

CA (Ex)

0%

20%

40%

60%

80%

100%


% R

esponse R

ate


Summary

Conclusion Effective push routing depends on worker expertise Concepts are effective for expertise profiling Combining task-assessment with self-assessment is

effective in reducing assessment cost

Future Directions Task routing under constraints

– Cost, Latency, Expertise, Utility Complex workflows in data quality management

21


Further Reading

U. Ul Hassan, S. O’Riain, and E. Curry, “Effects of Expertise Assessment on the Quality of Task Routing in Human Computation,” in 2nd International Workshop on Social Media for Crowdsourcing and Human Computation, 2013.

http://www.deri.ie/about/team/member/umair_ul_hassan/

22

2nd International Workshop on Social Media for Crowdsourcing and Human Computation

Paris, 1 May 2013

effects of expertise assessment on the quality of task routing in human computation

Technology

digital computation

expertise of human workers

workers workers

push routing workers

expertise profiles of

routing push routing

thequality of task routing

higherquality of task