analysing volunteer engagement in humanitarian mapping ... › staff › l.capra › publications...
TRANSCRIPT
Analysing volunteer engagement in humanitarian mapping:building contributor communities at large scaleMartin Dittus
ICRI Cities, UCLLondon, UK
Giovanni QuattroneICRI Cities, UCL
London, [email protected]
Licia CapraICRI Cities, UCL
London, [email protected]
ABSTRACTOrganisers of large-scale crowdsourcing initiatives not onlyneed to consider how to produce outcomes with their projectsbut also how to build a volunteer community. The initialproject experience of contributors plays an important rolein this, particularly when the contribution process requiressome degree of expertise. We propose three analytical di-mensions to assess first-time contributor engagement basedon readily available public data: cohort analysis, task anal-ysis, and observation of contributor performance and train-ing effects. We apply these to a large-scale study of remotemapping activities coordinated by the Humanitarian Open-StreetMap Team, a global volunteer effort with thousands ofcontributors. Our study yield substantive observations withdirect implications for coordination practices for all three an-alytical dimensions. We close by providing recommendationsabout how to build and sustain volunteer capacity in these andcomparable crowdsourcing systems.
Author KeywordsCrowdsourcing, volunteered geographic information,humanitarian mapping, peer production, social computing,enrolment, retention, engagement, task design, task analysis,training effects
ACM Classification KeywordsH.5.3. Group and Organization Interfaces: Computer-supported cooperative work; Design
INTRODUCTIONThe Humanitarian OpenStreetMap Team (HOT) aims to mapall the undocumented and crisis-stricken regions of the world.The formidable scale of this ambition was illustrated duringan activation in the context of the 2014 Ebola epidemic: evenafter months of work by thousands of volunteers, the newmaps of Central and West Africa are still not complete. An ar-ticle by Medecins Sans Frontieres (MSF) illustrates the scaleof the HOT challenge: to reach their goal, HOT organisersneed to grow their project to “the biggest instance of digital
Under submission to CSCW 2016, to be published in January 2016.
volunteerism the world has ever seen” [11]. Organisers thusnot only need to consider how to produce these maps, butalso how to foster a large global volunteer community in theprocess.
The HOT projects presented in this study represent two as-pects of this community-building challenge: disaster aid ini-tiatives need to build volunteer capacity to provide quickemergency response, and disaster preparedness initiativesneed to sustain volunteer capacity in the absence of urgentcauses. While organisers have significant freedom in design-ing these projects, it is not clear how to they can evaluatetheir choices in these regards. Furthermore it is not alwaysclear whether certain design choices may involve trade-offs.
Other studies have already assessed the quality of HOT out-puts, and their impact on the map [9, 33]. This study will in-stead focus entirely on engagement aspects: the existence ofHOT presents a rare opportunity to compare different coordi-nation practices within the same platform, involving a largenumber of projects and participants.
Proposed contributionsThe present study is focused on a key growth challenge: todevelop understanding of how best to increase volunteer ca-pacity. It takes the form of a large-scale quantitative obser-vational study. We evaluate whether individual projects cansuccessfully activate new volunteers (enrolment), but impor-tantly also retain them over time (retention). Together wedefine these as engagement.
A range of HOT initiatives and organisational practices of-fer many opportunities to evaluate specific organiser choices.We aim to assess a large number of participations in a con-sistent manner. To this purpose we propose three analyticaldimensions: cohort analysis where we compare collectionsof similar projects, task analysis where we compare projectsin their task complexity, and observation of performance andtraining effects relating to the rate of contributions. All relyon readily available public data, and we will demonstrate thatthey can yield important findings.
The analytical dimensions we propose are grounded in ex-isting theory, and have direct operational implications so thatfindings can be translated to organisational change. They pro-vide minimum-effort complements to more invasive evalua-tion practices such as controlled experiments, A/B tests andparticipant observations. They are general enough to be trans-ferrable to other online communities: their minimum require-
1
ment is a capacity to observe individual contributions overtime.
A key aspect of this evaluation is our focus on first-time con-tributors, starting from their initial enrolment. In these firsthours and days, we can expect that engagement is at least inpart shaped by the specific design of this first project. Wecan assess whether the initial experience was so discouragingthat contributors never returned, or whether it prepared themto contribute for longer periods. A more experienced con-tributor on the other hand may still be able to contribute to abadly designed task, which would make it harder to identifyproblematic design choices.
On the following pages we first present three research ques-tions motivated by the growth challenge we presented. Wethen provide an introduction to HOT and its practices, de-scribe key project initiatives, present an overview of relatedwork, and introduce our methodology. Finally we address ourresearch questions with a set of analyses based on contribu-tor engagement metrics, and close with a discussion of ourfindings, and a brief outline of future work.
RESEARCH QUESTIONSRQ1: Cohort analysisAre different coordination practices associated with differentcontributor engagement characteristics? In large distributedonline communities we are likely to find different subgroupswith divergent practices. In the case of HOT we find thesein the form of different humanitarian causes within the sameplatform.
We expect that divergent coordination practices and circum-stances can affect contributor engagement in different ways.In particular we believe that the perceived urgency of a causecan act as an attractor and lead to higher enrolment. On theother hand we expect that sustained promotion of needs canlikely increase retention, however it is unclear how well thisactually works and how long it can last.
A comparison of the engagement characteristics of differentknown cohorts can help reveal the impact of such coordina-tion choices and circumstances. Such a comparison can helpidentify where to look for successful coordination practices.In some cases any differences in outcome may be a resultof known circumstances or well-documented practices, and itmay be possible to transfer these to other contexts. In othercases it may require follow-up studies to establish why certaindifferences occurred.
RQ2: Task analysisAre different task designs associated with different contribu-tor engagement characteristics? Task analysis serves to as-sess the impact that contribution mechanics can have on theinitial enrolment experience. It allows to distinguish betweendifferent tasks in terms of their task complexity.
We expect that minimum degree of task complexity is likelyneeded to yield substantial contributions, however there maybe diminishing returns: more complex work can be discour-aging. This may be particularly true for newcomers, who maychoose to abandon their participation early.
We further expect that such effects can be addressed with bet-ter contributor guidance. A gradual learning curve and otherforms of guidance can help newcomers build a sense of self-efficacy. However we also expect diminishing reports in thiscontext: too much documentation can be overwhelming aswell.
Task analysis can help identify specific characteristics of a“complex” task. It can be used to assess how complex a taskshould be, and how much guidance should be provided, be-fore it becomes overwhelming. Follow-up studies can thenbe used to evaluate specific task design details, for examplecombinations of task properties, and specific forms of guid-ance.
RQ3: Contributor performance and training effectsIs contributor performance an early indicator of increasedretention? The rate of contributions of a contributor, or editpace, can be considered a key measure of their performance.Do faster contributors tend to stay longer? Is there a perfor-mance improvement during the initial period of participation?Are any such training effects associated with increased reten-tion?
We expect that faster contributors tend to stay longer, theymay be more confident in their abilities. We further expectto see training effects for OSM newcomers who may startslowly but will pick up pace quickly. We do not expect tosee similar training effects for contributors with prior OSMexperience.
To the extent that training effects can be observed we expectthat they are associated with increased retention, as a resultof increased self-efficacy and enjoyment. However the effectmay be limited to short-term retention, not necessarily long-term retention.
If such retention effects can be found we expect that trainingeffects can be used as an early engagement indicator withouta need for long-term observation. The presence of trainingeffects would also indicate a need for organisers to understandtraining as a key part of community building, to understandwhat kind of training takes place, and to then increase trainingopportunities.
REMOTE MAPPING WITH HOT AND OSMHOT started in 2010 as an informal network of experts andcommunity organisers, and the organisation has gradually re-fined the necessary processes and technologies that allow itto scale [29]. It coordinated responses to typhoon Haiyan in2013, the West African Ebola crisis in 2014, the 2015 Nepalearthquake, and many other activations [31]. In late 2014several partnering organisations launched a “Missing Maps”initiative where the focus is on proactive mapping to ensureplaces are already well-documented before a crisis hits [27].
The promise of these initiatives is to achieve a greater vol-unteer capacity for disaster response and humanitarian aidby incorporating the help of a global online volunteer force.The MSF staff member Ivan Gayton summarises the partic-ular appeal HOT has for representatives of aid organisations:
2
“Finally, I can give volunteers something to do that isnt justgiving money” [18].
The maps produced by HOT volunteers are free for all undera liberal license, and are now in use by experts at MSF, theAmerican and British Red Cross, the World Health Organisa-tion, and a growing number of other institutions [7]. A 2014report by MSF discusses the impact such initiatives can haveon the work of aid organisations: “Many interviewees com-mented that they were amazed by the speed at which the areawas mapped with the help of the volunteers. On his own, theGIS officer would not have been able to produce these basemaps during his mission.” [17].
HOT organisers coordinate a range of activities within an ac-tivation. They may produce project designs to address par-ticular information needs, and promote these to potential vol-unteers. Projects seek to map certain geographic features ina particular region, typically for a specific purpose: for ex-ample to map settlements so that aid experts can understandpopulation distributions, or to trace roads so that field teamscan plan transport routes. Larger HOT activations can consistof dozens of such individual projects.
Typically the first step in the creation of a new map is a re-mote mapping practice involving the help of hundreds or eventhousands of online contributors. This practice forms the sub-ject of our study, and the contribution mechanics is describedbelow.
Remote mapping with the tasking managerThe role of volunteers in remote mapping is to trace mapsfrom satellite imagery. The HOT tasking manager is a keytechnology in the contribution process, it emerged out of aneed to distribute work across large numbers of remote map-pers while reducing edit conflicts [23].
The tasking manager is also a rich source of contextual in-formation: it publishes a list of all remote mapping projects,task descriptions, boundary polygons for project regions, andcontributor lists. Figure 1 shows a screenshot of a project andits description.
Within all tasking manager projects, work is divided spatiallyinto smaller tasks (map grid cells). Contributors choose afree task in the tasking manager when they begin their work,and then contribute according to a set of instructions. Allcontributions are made on OSM using OSM mapping tools,and HOT contributors require an OSM account to contribute.
Contributors start by creating an OSM account, and use it tosign up on the HOT tasking manager. They then contributeto the map using OSM editing tools such as JOSM or iD.Most of their time is spent in these editing tools rather thanthe tasking manager. In contrast to other forms of OSM con-tribution, HOT volunteers do not map their local area: theytrace satellite images of remote places which they are likelyunfamiliar with. Their OSM changesets are automatically an-notated with HOT-specific tags, which means HOT contribu-tions can be identified in the OSM edit history, however thiswas not always the case. Further background on OSM and itsrelationship to HOT is provided by Soden et al [29].
Figure 1. The HOT tasking manager, showing the general introductionto a project on the left hand side, and a map with a task grid on the right.
Figure 2 visualises the global distribution of all HOT edits.Remote mapping is typically just a first step: in subsequentstages local mappers with knowledge of the respective areamay augment the maps by adding more detailed annotations,including placenames and other details, and by making cor-rections based on ground observations [30].
For this study we focus on the remote mapping process, andwe will not assess the contributions of local mappers: thereis less less readily available data on the circumstances andoutcomes of their practices, and these subsequent activitieson the ground are not as easily observed at scale. As a resulta large-scale quantitative evaluation of their work is harderto achieve. Their engagement characteristics are also likelydifferent from those of remote mappers: they involve a muchsmaller number of contributors who are often more closelyembedded with the local community or an aid organisation.
RELATED WORKWork that aims to understand engagement challenges of thekind encountered here can be found in a number of contexts.The most immediate domain is the study of crowdsourcing,which is the solicitation of labour from a large group of par-ticipants, typically online, accomplishing a variety of taskssuch as the creation of content or solving of problems [13].This includes forms of citizen science where a coordinatingparty invites contributions by laypeople [34]. Crowdsourc-ing systems in which participants are motivated by paymentand similar incentives are sometimes distinguished as crowdwork [16].
In crowdsourcing and crowd work systems, organisers (or“requesters”) prepare tasks that are then offered to contrib-utors. These organisers need to strike a balance between or-ganisational performance and worker satisfaction: “design-ing tasks suited to online participation that will elicit valuablecontributions while maintaining volunteers interest” [34]. Inother words, badly designed tasks can be deterrents for par-ticipation. A basic example of a design strategy is to splitthe work into smaller pieces that are manageable by a singlecontributor in a limited amount of time.
3
Figure 2. Geographic distribution of all HOT contributions. Many contributions are in central and west Africa, but also south-east Asia, and otherparts of the world.
Kittur et al. identify this as one of the key challenges ly-ing ahead for crowdsourcing system designers: “both weand our surveyed workers have observed many cases wherepoor quality work arises from poorly designed crowdsourcingtasks”. They propose to improve task designs by communi-cating more clearly with contributors, but also by facilitatingtheir learning experience: “Workers may need to acquire newskills to perform unfamiliar tasks, before or in the midst ofperforming the actual work” [16].
There is some evidence that increased activity and increasedretention may not always be achievable at the same time. Inonline citizen science projects it was found that more prolificcontributors can have shorter retention periods [25, 26]. Asimilar effect was found for crowdsourcing designs that aimto increase member productivity, and was attributed to eitherburnout or a sense of a “mission accomplished” [32].
HOT emerged out of the activities of the OSM community,and is shaped by the technologies and practices of that com-munity [23]. In principle anyone can contribute, and contrib-utors choose their own tasks freely. For this reason it can alsobe considered an example of commons-based peer produc-tion [2]. A substantial amount of research has studied peerproduction systems, with Wikipedia and OSM as prominentexamples. However in such systems there typically is no cleardistinction between organiser and contributor, while in HOTa central board and activation committee coordinate activi-ties with aid organisations. As a result, work in HOT may beorganised differently than in OSM or Wikipedia, and contri-bution processes tend to be more formal and goal-oriented.
A key barrier to entry for first-time HOT contributors is thefact that mapping with OSM is a complex practice: it re-quires specialist tools and an understanding of specialist con-cepts [28]. To our knowledge there is no published researchon how this affects HOT contributor engagement, however
there is some knowledge in related domains. The relativecomplexity of requested work was shown to affect contrib-utor engagement. In an evaluation of Mechanical Turk work-ers, Khanna et al. found that more complex tasks that requirea nuanced understanding of the domain can pose barriers toparticipation. A further key barrier was the relative complex-ity of instructions. Other barriers included problems relatedto the user interface, or misunderstandings derived from dif-ferences in cultural contexts between coordinator and contrib-utor [15].
A basic psychological model that allows us to reasonabout the nature of such barriers is the framework of self-efficacy [1]. Bandura proposed that perceived human effi-cacy determines if an individual will initiate an activity, howmuch effort will be expended, and how long the activity willbe sustained. In this model, self-efficacy is derived fromfour principal information sources: performance accomplish-ments, vicarious experience, verbal persuasion, and physio-logical states. In other words, tasks designs that aim to in-crease self-efficacy must consider how best to bring about ex-periences of mastery, and to not overwhelm prematurely. Thiscan at least in part also involve social processes: persuasion,observation of others, and feedback.
When designing tasks, one should also consider the motiva-tions of contributors to participate. Both intrinsic and ex-trinsic motivations can become drivers for participation whenthey are incorporated appropriately in project designs.
Shervish (1997): factors that affect social participation (vol-unteering) and charitable giving - strongest factors: presenceof a community of participation, stable beliefs that connectwith the charitable cause - perceived social urgency plays arole as well
4
A general model of motivations of volunteer workers was pre-sented by Clary et al. who describe six basic motivationalcategories, including values (such as altruism), social expe-rience, and enhancement (self-improvement, a positive self-image) [8]. In a study of Wikipedia contributor motivations,this was later amended by two further categories: fun, andideology [19]. Among early OSM contributors, Budhathokifound that an individual’s local geographic knowledge wasthe most significant driver to contribute [3].
Research gapsWe present the first comparative study of HOT activity acrossmultiple large initiatives, and likely the first large-scale quan-titative study of HOT community engagement. Despite in-creased research interest in the topic, to our knowledge thereis no published knowledge on the ability of different HOTproject designs to build volunteer capacity, and then suc-cessfully retain trained contributors over longer periods. Wepresent such a study: a quantitative evaluation of the impactof different modes of organisation and task designs on con-tributor engagement.
METHODOLOGY
DataAll our analyses are based on two data sets:
1. Project information published on the HOT tasking man-ager.1 This data was scraped for every project.
2. The OSM edit history of all map contributions, recordingthe creation and modification of all map objects over time.This dataset is freely available for download.2
Using contributor lists from the tasking manager as a start-ing point, we extracted the OSM map contributions by allknown HOT participants and cross-referenced them withHOT projects based on username, date, and location. For thepurpose of this study, any creation or modification of a mapobject is considered an edit.
Study periodWe selected an 18-month enrolment window from mid-2013to late 2014 to observe first-time contributions, further ex-tended by a buffer period of 180 days for the observation ofcontributor retention. This time captures the most active pe-riod of the tasking manager history to date. It excludes aninitial early-adopter period, but includes the first tasking man-ager use at a larger scale in late 2013 [23].
Figure 3 shows the remarkable growth of HOT remote map-ping activity in this time: by early 2015, HOT organisers hadcreated almost 1,000 remote mapping projects. Our study pe-riod is highlighted in the graph.
The specific timeframe, all dates are inclusive:
• First date of enrolment window: 16th of June 2013• Last date of enrolment window: 15th of December 2014• Plus 180-day buffer period: ends 16th of June 20151http://tasks.hotosm.org2http://planet.osm.org/planet/full-history/
Jan 2013
Jan 2014
Jan 20150
100
200
300
400
500
600
700
800
900
Num
ber
of
HO
T p
roje
cts
Figure 3. Growth of HOT tasking manager projects over time. Ourstudy period is highlighted.
Cohort selectionFor this first study of HOT engagement we limited our eval-uation to key remote mapping initiatives: larger collectionsof tasking manager projects with a shared singular purpose.These can be official HOT activations, or other large initia-tives. Our aim was to identify collections of projects whichare similar in terms of purpose and organisational practice.We identified such collections based on project listings on theHOT homepage and the OSM wiki. Approximately 50% ofHOT projects could not easily be allocated to a group.
In order to find collections that are suitable for both cohortanalysis and task analysis we further rejected initiatives thatinvolved less than 50 contributors, and less than 10 projects.We identified six larger initiatives as candidates. Of those wechose three as study cohorts based on their coordination pro-files: they represent a cross-section of typical HOT coordina-tion practices, from urgent responsive mapping to sustainedproactive mapping. Their aggregate size is substantial: theyencompass 30% of all projects on the tasking manager, andaccount for almost 50% of all first-time contributors in theperiod.
Typhoon Haiyan (cohort: TH)Also known as Typhoon Yolanda, TH was a tropical cyclonethat devastated the Philippines on November 8, 2013. TheHOT projects associated with it were of high urgency: themap data was used in disaster response and humanitarian aidas soon as it became available. TH was probably the firsthighly promoted HOT initiative to rely on the tasking man-ager for coordination. Volunteer work started only few daysbefore the event, initially with projects to prepare a base map.The focus later switched to damage assessment of the affectedareas. The OSM wiki lists 22 mapathons which were organ-ised around TH in November 2013 in cities around the world.Typically these were one-off events [22].
Ebola response (cohort: ER)The Ebola outbreak began in Guinea in early 2014. Aid or-ganisations needed maps to locate and treat those infected,yet many of the affected areas were not documented on any
5
Cohort Projects First-time contributors
TH 23 481ER 65 881MM 11 208
Total 99 1,570Table 1. Project cohort sizes.
existing maps. Initially this was treated as an urgent one-off event, and activity stopped soon. However as the epi-demic spread to neighbouring countries coordination waspicked up again, and the strategy changed to a more sus-tained effort covering larger regions. The initiative lasted un-til early 2015. Mapathons were organised in many parts ofthe world to train newcomers and coordinate volunteers, in-cluding monthly events in a growing number of cities [20].Project descriptions indicate that many activities were coor-dinated by representatives of aid organisations.
Missing Maps (cohort: MM)This proactive effort launched in November 2014: regionsvulnerable to crises are mapped early so that maps are alreadyavailable when a crisis occurs. In contrast to TH, this initia-tive is proactive rather than reactive, and continuity of con-tributor engagement is more important than a quick response.Where TH and ER were more ad hoc in their coordination,MM organisers set up structures to support more sustainedengagement: there are regular mapathons in many parts ofthe world, often organised as monthly events. A signup formfor volunteers allows contributors to get informed of upcom-ing initiatives and mapathons [21]. Similar to ER, many ofthe MM projects were initiated by representatives of aid or-ganisations.
We selected contributions relating to each cohort in the studyperiod, identifying instances where new first-time contribu-tors joined a project. The final data set derived from this issummarised in Table 1, with a total of 1,570 first-time con-tributors across 99 projects.
A timeline of contributor enrolments per cohort is shown inFigure 4, this plot illustrates a number of key aspects of ourdata set. The cohorts were active at different periods: TH inlate 2013, ER throughout much of 2014, and MM from late2014 onwards. The contributor distribution also indicates thatmany first-time contributors already had existing OSM useraccounts, as indicated by their user IDs. We will investigatethe distribution of prior OSM experience per cohort in a latersection. Finally the plot reveals a participation gap for ERbetween May and July 2014, this reflects the aforementionedperiod of inactivity between its initial activation and conclu-sion in early 2014, and a subsequent resurgence of activitylater in the summer.
Contributor observation periodsWe collected all contributions by first-time contributors in thefirst 180 days after their initial contribution. We paid particu-lar attention to three different timeframes per contributor:
Jan 2014
Jul 2014
Jan 2015
OSM
use
r ID
TH
ER
MM
Figure 4. Timeline of contributor enrolment, broken out per cohort.Contributors are arranged vertically in order of ascending user ID.
• The initial enrolment period of the first 48 hours. This pe-riod is used to observe initial contributions, and to assesshow many contributors returned immediately on the secondday. The short initial observation period of two days waschosen based on the median difference between the firstand last moment of contribution, which is only 20 hours.
• A 90-day retention period from the moment of enrolment.This was chosen based on an analysis of average contribu-tor lifetimes: almost 90% of contributors cease contribut-ing after 90 days of their initial participation.
• A 180-day survival period from the moment of enrolment,to identify the last known moment of contribution. Thiswas used for survival analysis: we considered contributors‘dead’ if they had been inactive for at least 90 days by theend of this survival period. The last known date of contri-bution before that point marks their ‘death event’.
In summary this means that we only consider the first 90 daysof contribution activity for our analyses, however we observefor a full 180 days after enrolment to establish abandonmentwith some degree of certainty.
Engagement metricsBased on these relative observation windows we computedengagement metrics for every first-time contributor we iden-tified. We first identified contribution sessions based on thetimestamps of individual contributions, with a session time-out of one hour. Based on these sessions we computed thenumber of labour hours spent on each contribution, using aprocess described by Geiger et al in the context of Wikipediacontributor analysis [12] .
A first set of engagement metrics are measures of activityin the enrolment period. We chose time-based activity mea-sures rather than simple edit counts because they allow us to
6
more meaningfully compare contributor effort across differ-ent kinds of tasks [12]. We seek to quantify the amount oftime spent contributing, rather than the volume of data thatwas produced, so we can compare contributor effort acrossdifferent map object types.
More specifically, we captured the number of labour hoursl48h in the first 48 hours. These are also calculated separatelyfor the first and second day in this initial period: ld1, ld2. Wefurther capture the rate of contributions in the firth 48 hoursc48h, measured in edits per hour, and contribution rates forthe first and second day cd1 and cd2. These allow us to de-termine a change in pace between the first and second day totest for the presence of training effects. This change in paceis described by the ratio cd2/cd1.
A second set of metrics are measures of retention. These arecalculated for each project that had first-time HOT contribu-tors: what share of contributors that joined HOT for the firsttime in this project were retained for further activities? Toquantify short-term retention per project we determine Rd2,the percentage of first-time contributors that are still activeon the second day of their participation, contributing to anyHOT project. Additionally we calculate long-term retentionmetrics Rm2 and Rm3, the percentage of first-time contributorsthat are still active in the second and third month. These 30-day periods were chosen to reflect monthly mapathon cyclesobserved by some HOT initiatives.
Quantifying prior domain experienceWe further quantified each contributor’s degree of prior OSMexperience as dpre, the number of days on which they had con-tributed to OSM before they joined their first HOT project. Incertain analyses we used this degree of prior OSM experienceas a control variable: contributors with prior OSM experiencemay find it easier to contribute to HOT.
In an initial assessment of the impact of prior experience wecorrelated dpre with our engagement measures. We found thatless experienced users contribute for less hours during theirenrolment (Spearman correlation coefficient ρS = 0.19), thatthey contribute more slowly at a smaller number of edits perhour (ρS = 0.23), and that are retained less often than othersin the second month (ρS = 0.12) and third month (ρS = 0.13,all with p < 0.001).
Comparison across cohorts shows that the three groups havedifferent constituencies: the most experienced group was TH(median: 5 days of prior OSM contributions, mean: 113days), the most inexperienced group was MM (median: 0days, mean: 23 days), and ER was in between the two (me-dian: 0 days, mean: 52 days). Table 2 visualises the distribu-tions as a histogram. Pairwise comparison of these distribu-tions with Kolmogorov-Smirnov statistic (KS) confirmed thatthese are statistically different populations (p < 0.001), withthe greatest difference measured between TH and MM (KSstatistic: α = 0.41).
Task analysisWe rely on the fundamental assumption that some tasks aremore challenging than others, and that this affects contribu-tor engagement. We expect that we can identify this effectby observing participation across different tasks. We specifi-cally aim to assess how easy it is for a newcomer to becomea productive contributor. What kinds of contributions needto be made? Does the task documentation provide sufficientguidance?
In “Task Complexity: Definition of the Construct”, Woodpresents three essential task components that relate to taskcomplexity: products, (required) acts, and informationcues [35]. In other words, he distinguishes between the com-plexity of the required work itself, and the amount of infor-mation cues and guidance necessary to produce these. Thesecan be quantified as number of acts and number of infor-mation cues. Both are measures of task complexity: simpletasks require processing fewer cues than complex tasks [24].They reflect the consideration that task designs must considerhow best to bring about experiences of mastery without beingoverwhelming.
Table 3 lists the six task design features we considered forthis study, including motivational factors, visual complexityof satellite imagery, task complexity, and forms of guidance.These were labelled by the first author in an iterative pro-cess, involving a detailed study of a large number of projects.We then counted the distinct number of map object types re-quired to fulfil a task. To this purpose we broadly categorisedall requested map features by their geometries and semanticrole: natural features, roads and highways, settlement bound-aries, buildings, urban infrastructure, and other features. Wesimilarly classified and counted the distinct number of infor-mation cues per project: stated priorities, descriptions of mapobject types, explicit sequences of work steps, and externallinks to coordination pages, reference documents, or instruc-tion manuals. Each of the features considered for this anal-ysis was used in at least 10 projects, other less widely usedfeatures were classified as ‘other’.
These low-level features were then used to compute task de-sign features as described in Table 3, these comprised our fea-ture vectors for each project. We standardised these featurevectors using z-scores, so that all variables have a mean of 0and a standard deviation of 1. We found no multicollinearityacross the standardised variables: the condition index of allvariables is 2.33, and we found no near-zero eigenvalues intheir cross-correlation matrix.
FINDINGS
RQ1: Cohort analysisTable 4 shows median engagement statistics per cohort for theenrolment period of the first 48 hours. TH and ER have sim-ilar enrolment profiles, while MM contributors appeared towork longer but also contributed more slowly than the othercohorts. We confirmed the pairwise difference of these distri-butions with a KS statistic: with one exception these differ-ences in distribution were statistically significant (p < 0.01).The only instance where a difference could not be asserted
7
Experience dpre 0 days 1-9 days 10-99 days ≥100 days
TH 30.9% 22.2% 20.3% 26.6%ER 52.8% 24.3% 11.7% 11.2%
MM 72.8% 18.4% 5.3% 3.4%Table 2. Distribution of prior OSM experience per cohort, in days with contributions (dpre).
Aspect Variable Description
Motivation has context Does the project description state an explicit purpose?Visual complexity urban density Is the mapped region rural (simple), mixed, or urban (complex)?Task complexity num concepts What is the number of requested map object types?Task complexity building trace Are buildings to be mapped as points (simple) or polygons (complex)?Guidance num cues Number of information cues provided in the documentation?Guidance num tag ex Number of tag examples listed?
Table 3. Task design feature vector produced by our task analysis.
Cohort l48h c48h
TH 1.14 640.4ER 1.16 620.0MM 1.29 529.2
Table 4. Median contribution activity by cohort: labour hours (l48h) andcontribution rate (c48h) in the first 48 hours.
were the distribution of labour hours in the TH and ER co-horts, however their rate of contributions differed.
Table 5 shows the corresponding median retention rates percohort, broken out for different short- and long-term retentionperiods. These measures indicate that TH and ER have thehighest short-term retention, with almost 30% of contributorsreturning on day 2. However TH then experiences a steepdrop in retention, its long-term retention rates are the lowestof all cohorts. In contrast MM starts with the lowest short-term retention of only 10% returning on second day, followedby the most stable long-term retention of all cohorts: almost9% contributors are still retained by month three.
We computed survival functions for each cohort based on aKaplan-Meier estimate with a 95% confidence interval. Apairwise logrank test confirmed statistically significant dif-ferences in survival rates between ER and TH (p < 0.001),no other pairing was found significant.
When limiting survival analysis to OSM newcomers only,where the first HOT contribution was also the first OSM edit,we observe the same trends but more pronounced differencesbetween cohorts. For these participants we observed a sta-tistically significant difference in survival rates between THand MM, as well as TH and ER (both p < 0.001). The cor-responding survival plot for OSM newcomers in Figure 5 il-lustrates these trends: TH has the lowest overall retention.ER and MM differ in the short term, with MM having lowerinitial retention, yet then achieve similar long-term retentioncharacteristics. According to the plot MM may even have thehighest long-term retention, yet this difference was not con-firmed by the logrank test.
Cohort Rd2 Rm2 Rm3
TH 26.8% 4.2% 4.6%ER 27.2% 13.6% 8.9%MM 10.1% 9.6% 8.7%
Table 5. Median retention for day 2 (Rd2), and months 2 and 3 (Rm2 andRm3). These indicate the percentage of HOT contributors who are stillactive in the respective period.
0 10 20 30 40 50 60 70 80 90
Number of days since initial contribution
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Surv
ival ra
te
TH
ER
MM
Figure 5. Survival functions for each cohort, indicating the rate at whichparticipants with no prior OSM experience ceased contributing. Shadedregions show 95% confidence intervals.
8
Variable Description
l48h Response: labour hours in first 48 hours.
has context Motivation: explicit purpose stated?urban density Visual complexity: rural or urban?num concepts Task complexity: # object typesbuilding trace Task complexity: points or polygons?num cues Guidance: # information cuesnum tag ex Guidance: # tag examples
dpre Control: prior OSM experience in days.num tasks Control: project size, in number of tasks.
Table 6. Regression features for RQ2. Task design features quantifydifferent aspects of task complexity and guidance.
RQ2: Task analysisWe prepared a regression model to observe any impact of taskdesign on early abandonment, measured in labour hours l48has the dependent. The regression model further contains thesix task design features of each contributor’s initial projectas independents. We included two control variables: prioruser experience dpre, and the size of the project (its number oftasks) to account for goal-setting effects with larger projects.The full set of regression features is listed in Table 6.
We found no good fit when regressing across all 1,570 sam-ples, the adjusted R2 scores were all below 0.1 for differentlinear regression methods. When applying the same regres-sion models for each cohort separately we found the samebad fit as before for the TH and ER cohorts. However a betterfit is found for the MM cohort and l48h when using ordinaryleast squares regression, with an adjusted R2=0.37, and an F-statistic of 18.72 (p < 0.001 at 7 degrees of freedom).
For this last model of MM task design impact there were twohighly significant regression coefficients (p < 0.001):
• Prior OSM experience per contributor: βdpre =0.5
• Number of map feature types requested per task:βnum concepts=-0.15, with a 95% confidence interval between-0.232 and -0.064
These coefficients are based on standardised z-scores. Thenegative coefficient βnum concepts indicates that strongest re-sponse is below the mean value of num concepts, which be-fore standardisation is at 2.7. This result suggests that taskdesigns of less than three distinct map features had strongerenrolment outcomes for this group: people contributed forlonger.
Other regression models yielded no improvements in fit.This includes models that only included OSM newcom-ers, or only involved the two significant features dpre andnum f eatureconcepts. The contribution rate c48h was not ex-plained by any regression model.
RQ3: Contributor performance and training effectsWe found that faster contributors tend to remain active forlonger: a correlation analysis found that the initial contribu-tion rate cd1 is associated with longer participation in the en-
0 200 400 600 800 1000 1200 1400
Rate of contributions
<0.5h
0.5-2h
>2h
Labour
hours
Figure 6. Rate of contributions (c48h) for contributors who are active fora given number of labour hours in the first 48h.
rolment period (Spearman correlation coefficient ρS = 0.15),and with increased retention in month 2 (ρS = 0.15, both withp < 0.01), however this effect is slightly reduced in month 3(ρS = 0.11, p < 0.05).
During a further analysis we found that the relationship be-tween initial contributor activity and the rate of contribu-tions follows a U-shaped curve: contributors who left earlyand those who stayed the longest tended to contribute morequickly than those in between the extremes. To further assessthis effect we segmented contributors into three engagementclasses based on the statistical distribution of l48h, the amountof labour hours they invest in the first two days. This vari-able follows a normal distribution. We segmented contribu-tors at the 25th and 75th percentile which yields three groupsof short-term (under 30 minutes), average, and long-term con-tributors (2 hours or more).
Figure 6 shows the distribution of contribution rates for eachengagement class. These reveal that first-time contributorswho left within the first 30 minutes also tended to have higherrates of contributions: the median contribution rate of thesecontributors (650 edits per hour) is almost 30% higher thanthat of participants who remained active for a full hour ormore (515 edits per hour). This effect was found in all co-horts. A pairwise comparison across the three bins with a KSstatistic over the respective distributions of l48h was signifi-cant for all pairs (p < 0.001).
On the other hand it cannot be said that there is a clear train-ing effect within the first sessions: the median change in paceis around 0% between the first and second day of enrolmentacross cohorts. We found no significant correlation betweenprior experience and performance improvements. Further-more we found no correlation between performance improve-ments and retention in month two or three: training effects donot appear to be associated with higher retention rates.
DISCUSSIONIn the following section we will summarise our results byhighlighting the key effects we observed, and close with aset of implications and design recommendations.
The three cohorts have different volunteer constituencies.A comparison of their distributions of prior experience in-dicated that cohort analysis of HOT contributors is worth-while: the three initiatives have markedly different distribu-tions. This may be a result of a difference in engagementstrategies: TH required a quick response by an existing com-
9
munity, while ER and MM could build volunteer capacity ina more sustained manner over longer periods. Furthermorethey were active at different periods: by the time ER and MMwere active HOT had already gained some prominence out-side of the OSM community.
The three cohorts have different contribution profiles duringenrolment. The initial contribution profiles of TH and ER arequite similar, although TH contributors contribute at a slightlyhigher pace. In contrast, MM volunteers have the lowest rateof contributions, yet are working longer hours during initialenrolment than the other cohorts. The lower pace may be aneffect of relative inexperience, while increased initial activitymay indicate a greater motivation to contribute.
Coordination practices have a marked impact on contribu-tor retention. Long-term retention was highest for the ERand MM cohorts which were specifically set up as more sus-tained initiatives, and which relied on a range of volunteerengagement practices such as mapathons and social mediause. It was further found that for the MM cohort of projects,short-term retention was lowest yet long-term retention washighest: contributors do not tend to come back on the secondday, however they are more likely to remain active a month ortwo later. This may indicate a greater reliance on mapathons,where contributors do not necessarily return to the project aday later, but where they may return at the time of the nextmapathon event. In contrast to this TH had a large number ofcontributors yet the lowest retention rates, likely because ofan absence of any such sustained engagement practices. Itscontributors may have been attracted by a perceived urgencyaround the disaster event, however they were not successfullyretained.
Complex task designs can be a deterrent for certain contribu-tor groups: projects involving three or more map object typessaw shorter activity periods for the MM cohort. In otherwords, projects should only involve a very small number ofdistinct map object types at a time, or they risk demotivatingfirst-time contributors. We found no indication that this ef-fect is limited to contributors without prior OSM experience.We further did not identify such effects in the other cohorts,and found no evidence of an impact of documentation andguidance on engagement.
Most first-time HOT contributors tend to operate at a fairlysteady pace, however contributors with more prior OSM ex-perience tend to work faster and stay a little longer. This ef-fect is consistent across cohorts.
Early abandonment is associated with higher contributionrates. Volunteers who stopped contributing within the first30 minutes tended to contribute at a higher pace than thosewho stayed an hour longer or more. This may suggests an in-stance of a burnout effect, where some first-time contributorsbegin their engagement at a relatively high pace but then losemotivation quickly. The effect was found in all cohorts.
Training effects are not associated with increased retention:those contributors whose performance increased within theenrolment period were not necessarily retained for longer,
suggesting that the presence of training effects is not an earlyindicator of increased long-term engagement.
ImplicationsThe aim of our study is to understand engagement factors incrowdsourcing communities that help organisers reason abouthow to build and retain volunteer capacity. We believe wehave identified some key aspects that can inform the designand improvement of HOT and of related large-scale crowd-sourcing systems.
Our findings suggest that the capacity-building strategies ofthe ER and MM cohorts work well, and we encourage or-ganisers of other initiatives to adopt their practices: a com-bination of highly promoted projects over a sustained period,a steady stream of new efforts, regular mapathons and othertraining environments, and the use of email notifications inthe MM cohort as a means of notifying interested contributorsof new causes. In these two cohorts, a good share of contribu-tors kept coming back. In contrast to this, the greater urgencyof the TH initiative (a response to a discrete crisis event) mayhave attracted many volunteers, however it did not contributeto an increase in retention.
These retention effects may further be affected by the markeddifference in cohort constituents. According to project de-scriptions, both ER and MM initiatives were largely coor-dinated by representatives of aid organisations, while THprojects were coordinated by OSM community members.This may have affected how the initiatives were promoted,and to whom they were promoted to. It is possible that ERand MM participants already had existing connections to aidorganisations as part of other outreach efforts, and had a par-ticular motivation to remain engaged.
With one exception, the task designs encountered in our studywere found to be remarkably consistent in their engagementcharacteristics. One cohort saw a reduction in activity duringthe enrolment period for tasks that requested contributions ofa higher complexity, which suggests that organisers shouldlimit the number of map feature types requested per project.
Our finding of relative consistent task design performancemay also indicate a limitation of our observational study: wedo not compare radically different task designs, and insteadmerely observe the tasks that already exist, all of which relyon the same tools and interfaces for the actual contributionprocess. Additionally, many of these designs have alreadybeen informed by years of prior experience [23]. Conse-quently we want to encourage HOT organisers to also exper-iment with radically new task designs to identify alternativestrategies for further improvement. An example of this couldbe a micro-tasking interface that offers smaller and simplertasks, which could allow newcomers to become productivemore easily and quickly.
An alternative interpretation is that current HOT contributionmechanics do not have a major impact on engagement, andthat other aspects may be more important. In particular ourliterature review suggests that intrinsic participant motiva-tions, participant enjoyment, association with particular hu-manitarian causes, and social aspects of the HOT contribution
10
experience may be more important to contributor engagementthan the specifics of HOT contribution mechanics.
A further conclusion from our findings is that the best meansof increasing output capacity is to grow the volunteer base,particularly considering the vast amount of uncharted terri-tory that HOT aims to map. We could observe improve-ments in the performance of individual contributors, howeverbased on our observations an increase in contributors wouldincrease outputs more quickly.
It remains open what constitutes good training conditions forabsolute newcomers. We believe that given a choice, new-comers are best placed in projects where they have a higherlikelihood of being retained. In our case this would be the ERand particularly MM cohorts: projects that are specifically setup as long-term initiatives. Additionally there are indicationsthat particularly the MM cohort was successful at retainingand training absolute newcomers with no prior OSM experi-ence.
CONCLUSIONWe proposed an observational study to assess the relationshipbetween project designs and contributor engagement, with aspecific focus on the experience of first-time contributors. Wecompared project designs along three analytical dimensions:cohort analysis, task analysis, and observation of contribu-tor performance and training effects. Under consideration ofthese aspects we evaluated different projects in their ability tofoster contributor engagement in the short and long term. Theanalytical dimensions yielded plausible findings: we foundthat different coordination practices did have a marked en-gagement impact, and that differences in task design can havean impact for some groups. Additionally, we found that priordomain experience in first-time contributors is likely to in-crease their engagement.
We believe that these findings have some external validity:we encounter similar contributor engagement across differentcohorts, but also some differences, and we believe that mostof these differences have been explained by the observablefactors we discussed above. For this reason, we suggest thatour findings are transferrable to other kinds of crowdsourcingsystems.
The nature of this work and the available evidence howeverplace limits on our ability to build a more nuanced under-standing, and there are many aspects which we cannot gaugefrom the available data. For example the contributor contextis generally not known: in which situation does mapping takeplace? Yet the volume of projects and the number of partici-pants included in this study allow us to identify some generaltrends: aspects that tend to be shared across different kinds ofprojects.
Future workParts of our findings suggest that the participation setting mayplay an important role in contributor engagement: mapathonsand other social learning environments may play an impor-tant role in contributor onboarding. Although there is some
existing knowledge on these effects, we believe that their im-pact warrants further study. Some early observations weremade by Hristova et al. in a study of OSM mapping par-ties: the authors found that participants sustained engagementeven after the event, however this effect was limited to con-tributors with at least some prior experience, and newcomersto OSM could not be activated by these settings [14]. Otherstudies have found evidence that the socialisation experienceof first-time contributors in online communities can increasetheir contributions and long-term retention. In a recent obser-vational study of Wikipedia contributors, Ciampaglia et al.find that a successful early socialisation experience is asso-ciated with and can sometimes predict increased contributorengagement, however it was also found that the causal struc-ture between socialisation, motivation, and participation arenot entirely clear [6]. Further studies identified similar ef-fects [10, 4, 5].
Furthermore our analysis of task complexity could be aug-mented with further research involving other methods, suchas participant observation, usability studies, and ethnographicstudies, to better understand the actual contribution process.Such methods could asses which forms of guidance are mostuseful to newcomers, to what extent volunteers spend timereading task instructions before they start contributing, andrelated aspects of the contribution process.
We have not studied the impact of project designs on outputquality because it is a separate concern from that of contribu-tor engagement, but also since it would be significantly harderto measure. These maps are typically the first ever map oftheir kind, which means there is no ground truth data, anda manual quality assessment would require substantial effortat this scale. Additionally the mapped geographies tend tobe different across initiatives, which makes it hard to developintrinsic quality measures that have validity across differentregions of the world. HOT does have its own validation pro-cess, however it is highly informal and not sufficiently consis-tent for a rigorous evaluation. Yet there is an opportunity forfurther studies to assess how coordination practices affect thequality of contributions, and whether there are interactionsbetween quality concerns and contributor engagement.
This work could further be augmented with a more qualita-tive understanding of the contributor experience, by means ofparticipant observation or surveys. In particular, there is cur-rently no published knowledge on the motivations of HOTcontributors, and how they relate to those of other crowd-sourcing systems, including OSM. Many further engagementaspects remain unobserved, including the effects of partici-pant enjoyment, interactions between contributors, user inter-faces, and others, each worthy of a study of their own.
REFERENCES1. Albert Bandura. 1977. Self-efficacy: toward a unifying
theory of behavioral change. Psychological review 84, 2(1977), 191.
2. Yochai Benkler and Helen Nissenbaum. 2006.Commons-based peer production and virtue. Journal ofPolitical Philosophy 14, 4 (2006), 394–419.
11
3. Nama R Budhathoki. 2010. Participants’ motivations tocontribute geographic information in an onlinecommunity. Ph.D. Dissertation. University of Illinois atUrbana-Champaign.
4. Moira Burke, Cameron Marlow, and Thomas Lento.2009. Feed me: motivating newcomer contribution insocial network sites. In Proceedings of the SIGCHIConference on Human Factors in Computing Systems.ACM, 945–954.
5. Boreum Choi, Kira Alexander, Robert E Kraut, andJohn M Levine. 2010. Socialization tactics in Wikipediaand their effects. In Proceedings of the 2010 ACMconference on Computer supported cooperative work.ACM, 107–116.
6. Giovanni Luca Ciampaglia, Dario Taraborelli, and SanFrancisco. 2015. MoodBar: increasing new userretention in Wikipedia through lightweight socialization.In Proc. CSCW ’15. 734–742.
7. Liat Clark. 2014. The race to contain West Africa’sEbola outbreak - Wired UK. http://www.wired.co.uk/news/archive/2014-04/11/ebola-open-street-map.(2014).
8. E Gil Clary, Mark Snyder, Robert D Ridge, JohnCopeland, Arthur A Stukas, Julie Haugen, and PeterMiene. 1998. Understanding and assessing themotivations of volunteers: a functional approach.Journal of personality and social psychology 74, 6(1998), 1516.
9. Melanie Eckle and Joao Porto de Albuquerque. 2015.Quality assessment of remote mapping inOpenStreetMap for disaster management purposes.ISCRAM 2015 (2015).
10. Rosta Farzan, Robert Kraut, Aditya Pal, and JosephKonstan. 2012. Socializing volunteers in an onlinecommunity: a field experiment. In Proceedings of theACM 2012 conference on Computer SupportedCooperative Work. ACM, 325–334.
11. Medecins Sans Frontieres. 2014. MSF: the MissingMaps project. (2014).http://www.msf.org.uk/missing-maps-project.
12. R Stuart Geiger and Aaron Halfaker. 2013. Using editsessions to measure participation in Wikipedia. In Proc.CSCW ’13.
13. Jeff Howe. 2006. The rise of crowdsourcing. Wiredmagazine 14, 6 (2006), 1–4.
14. Desislava Hristova, Giovanni Quattrone, Afra Mashhadi,and Licia Capra. 2013. The life of the party: impact ofsocial mapping in OpenStreetMap. In Proc. ICWSM ’13.234–243.
15. Shashank Khanna, Aishwarya Ratan, James Davis, andWilliam Thies. 2010. Evaluating and improving theusability of Mechanical Turk for low-income workers inIndia. In ACM DEV. 1.
16. A Kittur, J Nickerson, and M Bernstein. 2013. Thefuture of crowd work. Proc. CSCW ’13 (2013), 1–17.
17. Timo Luge. 2014. GIS support for the MSF ebolaresponse in Guinea 2014. (2014). Medecins SansFrontieres, Logistics Department.http://preparecenter.org/resources/gis-support-msf-ebola-reponse-guinea-2014.
18. Chris Michael. 2014. Missing Maps: nothing less than ahuman genome project for cities - The Guardian.http://www.theguardian.com/cities/2014/oct/06/missing-maps-human-genome-project-unmapped-cities. (2014).
19. Oded Nov. 2007. What motivates Wikipedians?Commun. ACM 50, 11 (2007), 60–64.
20. OpenStreetMap. 2015a. 2014 West Africa EbolaResponse - OpenStreetMap Wiki.https://wiki.openstreetmap.org/wiki/2014 West Africa Ebola Response. (2015).
21. OpenStreetMap. 2015b. Missing Maps Project -OpenStreetMap Wiki. https://wiki.openstreetmap.org/wiki/Missing Maps Project. (2015).
22. OpenStreetMap. 2015c. Typhoon Haiyan -OpenStreetMap Wiki. https://wiki.openstreetmap.org/wiki/Typhoon Haiyan.(2015).
23. Leysia Palen, Robert Soden, T Jennings Anderson, andMario Barrenechea. 2015. Success & scale in adata-producing organization: the socio-technicalevolution of OpenStreetMap in response to humanitarianevents. In Proc. SIGCHI ’15. 4113–4122.
24. John W Payne. 1982. Contingent decision behavior.Psychological Bulletin 92, 2 (1982), 382.
25. Lesandro Ponciano and Francisco Brasileiro. 2014.Finding volunteers’ engagement profiles in humancomputation for citizen science projects. HumanComputation 1, 2 (2014), 1–17.
26. Lesandro Ponciano, Francisco Brasileiro, RobertSimpson, and Arfon Smith. 2014. Volunteers’engagement in human computation astronomy projects.Computing in Science & Engineering PP, December(2014), 1–1.
27. Missing Maps Project. 2015. Missing Maps - putting theworld’s vulnerable people on the map.http://www.missingmaps.org. (2015).
28. Manuela Schmidt, Silvia Klettner, and RenateSteinmann. 2013. Barriers for contributing to VGIprojects. In Proc. ICC ’13.
29. Robert Soden and Leysia Palen. 2014. Fromcrowdsourced mapping to community mapping: thepost-earthquake work of OpenStreetMap Haiti. (2014).
12
30. Sam Sturgis. 2014. Open-source mapping the world’smost vulnerable regions will save lives. (2014).http://www.citylab.com/design/2014/12/open-source-mapping-the-worlds-most-vulnerable-regions-will-save-lives/383337/
31. Humanitarian OpenStreetMap Team. 2015. FeaturedHOT projects. http://hotosm.org/projects. (2015).
32. Loxley Sijia Wang, Jilin Chen, Yuqing Ren, and JohnRiedl. 2012. Searching for the Goldilocks zone. In Proc.CSCW ’12. 989.
33. Clay Westrope, Robert Banick, and Mitch Levine. 2014.Groundtruthing OpenStreetMap building damageassessment. Procedia engineering 78 (2014), 29–39.
34. Andrea Wiggins and Kevin Crowston. 2011. Fromconservation to crowdsourcing: A typology of citizenscience. Proceedings of the Annual Hawaii InternationalConference on System Sciences (2011).
35. Robert E Wood. 1986. Task complexity: Definition ofthe construct. Organizational Behavior and HumanDecision Processes 37, 1 (1986), 60–82.
13