Computer Science Honours

Final Paper

2016

Title: Annotating the Veracity of Tweets through Mobile Crowdsourcing

Author: Shaheen Karodia

Project Abbreviation: SASITWIT

Supervisor(s): Selvas Mwanza and Dr Hussein Suleman

Category Min Max Chosen Requirement Analysis and Design 0 20 20

Theoretical Analysis 0 25 0

Experiment Design and Execution 0 20 5

System Development and Implementation 0 15 15

Results, Findings and Conclusion 10 20 10

Aim Formulation and Background Work 10 15 10

Quality of Paper Writing and Presentation 10 10

Quality of Deliverables 10 10

Overall General Project Evaluation (this section

allowed only with motivation letter from supervisor)

0 10 0

Total marks 80 80

DEPARTMENT OF COMPUTER SCIENCE

Annotating the Veracity of Tweets through MobileCrowdsourcing

Shaheen Karodia

University of Cape Town

Rondebosch

Cape Town, Western Cape 7700

krdsha003@myuct.ac.za

ABSTRACTTwitter is an established micro-blogging social network that allows

users to post short, 140 character messages, called tweets. The

quick propagation of information through the Twitter network

makes it conducive to opportunistic misuse such as Spam and non-

credible tweets, thus necessitating automatic detection of fake and

real tweets. This paper presents the design, implementation and

evaluation of a mobile crowdsourcing artifact and aims to assess

if crowdsourcing can be used to annotate the veracity of tweets.

The results from the final heuristic evaluation showed positive

feedback in general and indicated that the application conformswell

to Nielsen’s heuristics. However, it revealed 26 usability issues, the

majority of which have been corrected. Experimental results show

that the truth labels assigned through crowsdourcing performed

better than the truth labels assigned by the average annotator at

correctly identifying fake and real tweets, having achieved 13.14%

greater accuracy at identifying real tweet instances, and 21.15%

greater accuracy at identifying fake tweet instances.

CCS CONCEPTS• Information systems→Crowdsourcing; •Human-centeredcomputing → Social media; Mobile computing;

KEYWORDSTwitter, credibility, crowdsourcing, mobile phones

1 INTRODUCTION1.1 BackgroundTwitter is an established micro-blogging social network with in

excess of 328 million monthly active users since 20171. The so-

cial media platform is unique in that it allows for non-reciprocal

user connections where being followed by a user does not necessi-

tate having to follow back [19]. Twitter allows users to stay con-

nected through the exchange of short, 140 character, messages

called tweets, which can contain multimedia content in the form

of text, photographs, videos and Uniform Resource Locator (URL)

links.

When a user posts a tweet, it is propagated to their entire follower

network. In addition to this, users can search for tweets related to

0 This thesis was completed in partial fulfillment of a Bachelor of ScienceHonours degree in Computer Science at the University of Cape Town in 2017.An online appendix containing all additional resources can be found at https://people.cs.uct.ac.za/~knmkri002/ and will also be available from 12 Oct 2017 athttp://pubs.cs.uct.ac.za/ by searching for the SASITWIT project.1https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/

trending topics using the #topic_name function, they can reply and

mention other users with the @username directive and they can

retweet (effectively ’copying’ and forwarding another user’s tweet

with additional information)2. Retweeting allows for fast diffusion

of information throughout Twitter, exposing the tweet to users who

are not followers of the original poster [19].

The quick propagation of information through the twitter net-

work makes it conducive to opportunistic misuse. The forms of

abuse that have been studied and observed on the platform include

Spam such as malware, phishing and links to scams [14], in addition

to misinformation such as fake images [17], non-credible tweets

[15, 16] and topics [9], as well as political astroturfing (the practice

of making a political campaign appear as if it was started by the

public) [30].

Grier et al. showed that Twitter’s built in methods of malicious

content detection are too slow at protecting users, and that users

are highly susceptible to falling for Spam content on Twitter [14].

Furthermore, studies indicate that users are bad at assessing the

credibility of tweets [24]. Taken collectively, this highlights the

need for automatic real time credibility detection on Twitter.

1.2 Project Significance and AimsThis work forms part of a broader project to algorithmically detect

fake content on Twitter, which relies extensively on training data to

train a machine learning classifier. This paper proposes the use of

crowdsourcing, the process of outsourcing work to a large network

of individuals [31], to collaboratively label the training dataset.

In particular, crowdvoting will be used, whereby the answers of

collaborators will be aggregated into a single vote [29] to assign

truth labels to tweets and through the wisdom of crowds produce a

viable training dataset [8].

With this in mind, this work aims to address the following re-

search question: “Can crowdsourcing be used to annotate the ve-

racity of tweets?” Furthermore, it aims to describe the design, im-

plementation and usability of the mobile crowdsourcing artifact

produced.

1.3 Report StructureThis paper is organised as follows. Section 2 explores background

and related work. Design methodology and processes are discussed

in Section 3. In Section 4, the implementation of the crowdsourcing

mobile application is described. In Section 5 we focus on the eval-

uation of crowdsourcing in annotating the truthfulness of tweets.

Section 6 covers the outcomes of the usability evaluation of the

2https://support.twitter.com/articles/13920

software artifact. Sections 7 and 8 explore the project limitations

and ethical issues. Sections 9 and 10 conclude the paper and discuss

future work.

2 BACKGROUND AND RELATEDWORKThis work employs crowdsourcing as a means of data collection for

fake tweet detection on Twitter. As such, this section will discuss

previous work related to crowdsourcing as well as data collection

methods used in other Twitter misinformation detection studies.

2.1 Crowdsourcing2.1.1 Mobile Device Crowdsourcing. Various studies have uti-

lized crowdsourcing with mobile devices [2, 3, 5, 11, 12, 18, 23, 25,

27, 33, 36, 37]. Some leverage the rich feature set that mobile phones

offer for microtask creation, completion or allocation. These fea-

tures include the accelerometer, geolocation, camera and audio

capture, and will be referred to as feature specific crowdsourcing.Other studies rely mainly on the ubiquity of mobile phones and

can be described as feature independent crowdsourcing. Both feature

specific and feature independent crowdsourcing will be discussed

in this section.

Feature Specific Crowdsourcing. MCrowd specifically supports

sensor based tasks. These include camera dependent image collec-

tion tasks where users can ask for photographs to be taken and

submitted for particular landmarks [37]. Phone cameras have also

been used to automatically collect daily fuel prices from billboards

and petrol stations through computer vision [11].

Geolocation has been employed to allocate tasks [3] and visualize

data [33]. Microtasks can be pushed to users, whereby the task

allocation is a function of your current location, i.e. only tasks

in close proximity will be available to complete [3]. In contrast,

NoiseTube uses GPS location to map noise pollution from urban

areas by crowdsourcing sound from cellphone microphone sensors

[33].

In a similar vein, other crowdsourced events have also been

mapped using geolocation. Events such as road blocks, traffic con-

gestion and vehicle breakdowns can be logged by users to assist

other drivers in making decisions [2]. It can be further utilized to

reroute drivers on congestion free paths. Likewise, Nericell moni-

tors road conditions such as bumps, potholes, braking and hooting

through the phone’s accelerometer, microphone and GSM radio

[23].

Feature Independent Crowdsourcing. Txteagle provides a mobile

application that empowers users to earnMPESA and airtime through

the completion of small tasks [12]. These tasks include text tran-

scription, translation and survey completion. In a user study with

Nairobean security guards, high school students and taxi drivers,

all groups were able to complete a selection of translation tasks

with 75 percent accuracy. Similarly, both MClerk [18] and Mobile-

Works [25] make use of text transcription. MClerk pushes images

of words to users in order to convert language documents into a

digital format, with MobileWorks providing text transcription tasks

with the intent of allowing low earning users to supplement their

income through crowdsourcing.

Both SMSAssassin [36] and Ushadihi [27] rely on SMS based

tasks. SMSAssassin is a Symbian mobile based SMS spam filtering

application, that implements a SMS submission system to initialize

and keep their data set of spam SMS’s up to date [36]. In contrast,

Ushahidi allows users to report crisis related information such as

instances of violence through SMS [27].

Unlike the studies above that require human input, ComputingWhile Charging utilizes the inherent computational power of smart-

phones to perform work [5]. While phones are idle and charging

overnight, computational based tasks can be distributed to smart-

phones and aggregated once complete.

2.1.2 Crowdsourcing and Social Media. Crowdsourcing on so-

cial media has taken two forms: one, leveraging the data that users

post [4, 13], and two, building applications on top of the current

social media infrastructure [7, 21, 28, 32].

The former approach has been employed to annotate tweets

documenting users first-hand experience with adverse reaction

drugs [4]. This approach has further been used to scrape Spanish

tweets together with corresponding geotagging information, and

characterize the language usage into sub dialects and regions [13].

Many systems developed on top of social media require a call

for input from different user groups. An automated chat bot will

tweet an open call for response in the event that it does not have an

adequate reply to a given user prompt [7]. In contrast, LogicCrowd

relies on a call to a user’s Facebook friends to complete tasks such

as item recommendations, ranking of products and gathering of

crowd opinion [28].

Systems have been developed to visualise crowdsourced informa-

tion. UbiAsks allows tourists to make image based requests which

are then propagated to Japanese locals through Twitter [21]. These

Q and A’s are then depicted using Google Maps. Likewise, disaster

emergency information has been visualised using Google Maps

during the 2012 Haiti Earthquake [32]. Through Twitter, users are

asked to incorporate special hashtags into their tweets related to the

crisis at hand. These tweets are identified and processed through

Twitter’s search API and then plotted using Google Maps.

2.2 Data LabellingThis paper collectively forms part of a body of work to automatically

assess the veracity of tweets on Twitter. As such the data collected

will be used to train a machine learning classifier to algorithmically

determine the truthfulness of tweets. This section presents means

of data collection that have been used in other studies attempting

to automatically detect misinformation and abuse on the platform.

2.2.1 URL Blacklisting Classification. In a study aimed at char-

acterising the Spam content on Twitter, URL blacklist sites were

utilized to label Spam tweets [14]. The landing pages of URLs within

tweets were check against three blacklist sites, namely Google

Safebrowsing, URIBL and Joewein. Similarly, in labelling a subset

of Spam, namely phishing, URLs were checked against the Google-

Safebrowsing and PhishTank blacklists [1]. In conjunction with

this, URLs deemed malicious by Twitter’s built in detection system

as well as that of URL shorting services such as bit.ly, were also

labelled as spam. In contrast, WarningBird, makes use of Twitter’s

built in mechanism for detection of spam, by labelling URLs from

2

suspended accounts as malicious and those from active accounts as

non-spam [20].

2.2.2 External Sources Classification. 7 instances of verified

truths and 7 instances of verified lies related to the 2010 Chilean

earthquake were flagged through the use of sources independent

to Twitter [22]. Likewise, independent articles and blogs, and in

particular a list of fake and real images published by the Guardian

online website, were used to classify tweets containing images as

real or fake in relation to the Hurricane Sandy natural disaster [17].

2.2.3 Manual Classification. Many authors have sought to com-

pile datasets through manual labelling. Using the following two

heuristics: that spam bots have a small follower to followee ratio andthat spam posts primarily consists of links as opposed to personal posts,500 accounts were labelled as spam bots or legitimate accounts by

manually checking the 20 most recent tweets of each account [34].

The dataset was then supplemented by searching for posts where

users have commented@spam and manually verifying if they were

spam bots or not. A similar approach using the same heuristics was

employed to identify spam accounts [35]. URLs, username pattern

matching and keywords were utilized to classify tweets as spam

or not in connection with the trending topic #robotpickuplines [38].The authors of [30], in conjunction with additional volunteers, la-

belled memes as truthy or legitimate in relation to the 2010 US

political election. In the event there was no consensus determined,

disagreements were resolved through group discussion.

2.2.4 Crowdsourcing Classification. In assessing the credibility

of tweets related to trending topics, Mechanical Turk was used to

ask annotators first to label tweets as either news or general conver-

sation and of those items identified as news label the likelihood that

the messages were credible [9]. All labels were then assigned based

on majority vote. Mechanical Turk was further used to provide

credibility judgments for tweets on a 5 point likert scale [26]. Simi-

larly, making use of another crowdsourcing platform, CrowdFlower,

annotators were asked to label tweets related to 6 events of 2013

based on how credible they felt the tweets were [16].

3 DESIGN METHODOLOGYAn iterative, user centered design approach was followed through-

out the project. This methodology was selected to allow for con-

tinuous feedback to improve the design incrementally, and ensure

that the final software artifact met the needs of users. Users were

involved during each iteration in the form of individual interviews,

focus groups, paper and digital prototype testing. Based on feed-

back received from users, refinements were made to the system

functionality and user interface for subsequent iterations.

3.1 Iteration One3.1.1 Requirements Gathering. Initial requirements gathering

was conducted with four participants with high computer literacy

levels, and varying levels of social media familiarity, ranging from

not active on any social media platform to being active on multiple

social media platforms. The process involved individual interviews

with users, resulting in the following major requirements:

• Both the tweet content and identity of the poster must be

displayed during the annotation process

• Users must be able to see their annotation contribution

relative to other users in the form of a leaderboard

• Users must be able to rate the tweet on a credibility scale

Many interviewees claimed that they do not like gamification as

an incentive so its implementation has been left out of scope.

3.1.2 Technological Feasibility. Anminimum viable product was

initially developed to test the feasibility of the technologies selected

for development. A single page Ionic/Cordova application was built

with the following functionality:

• Rendering a tweet in the user interface using Twitter’s

JavaScript tweet widget3

• Connecting, writing and reading from Google Firebase’s

Real Time Database4

3.2 Iteration Two3.2.1 Focus Group. A focus group with four Computer Science

Honours students was conducted to discuss the implementation

and build on the requirements identified in iteration one. The ma-

jor outcomes relate in particular to the process of annotation. The

discussion elicited that the tweet must be rendered the same way as

user would see it through using Twitter. Furthermore, users would

like to be able to skip tweets that are irrelevant, or that cannot have

a rating assigned e.g. tweets expressing non news related conversa-

tion. With respect to the rating scale, all participants agreed that

options presented should be a word based scale from highly un-

likely to be true to highly likely to be true, as opposed to a numeric

or smiley face scale in order to remove ambiguity.

From these requirements, and those identified in iteration one, a

use case diagram was made to map all goals that users can achieve

with the system. Furthermore, the 3 most complex use cases were

fledged out in use case narratives for better understanding. Both

the use case narratives and use case diagram can be found in the

online appendix5.

3.2.2 Paper Prototyping. Subsequently, a paper prototype of

the crowdsourcing application was developed with the intention

of gaining early feedback on flow as well as learnability of the

system. The features presented in the prototype are based of the

requirements identified through the interviews and focus groups.

The prototype was tested using 3 undergraduate Computer Science

students. The wizard of Oz and think aloud techniques were em-

ployed with users being asked to verbalize their thought process

while completing a series of tasks using the paper prototype, while

I performed all actions as the "computer". Users were informally

interviewed afterwards and both their pain points and positive feed-

back were noted. Major usability issues expressed by participants

were related to difficulty navigating the application through the

use of home and back buttons, and the ambiguity in the meaning of

elements on both the annotation and leaderboard pages. All partic-

ipants suggested a more comprehensive and visual based tutorial.

A paper prototype of the Label Tweets page (Annotation page) can

be seen in Figure 1a.

3https://dev.twitter.com/web/embedded-tweets

4https://firebase.google.com/docs/database/

5https://people.cs.uct.ac.za/~knmkri002/

3

(a) Paper Prototype (b) Digital Prototype (c) Software Implementation

Figure 1: Iterations of the Label Tweets Screen

3.3 Iteration Three: Digital PrototypeFeedback from the 2nd iteration was incorporated into the digital

prototype by changing the navigation to a side menu and hav-

ing more comprehensive explanations of the app elements in the

visual, slide based tutorial. The digital interactive prototype was

developed using Proto.IO6. The tool was selected since it had com-

ponents prestyled using Google’s Material Design specification7.

Furthermore, the prototypes developed can be previewed on An-

droid devices, thus giving users the most realistic representation of

how the app would operate on a real mobile phone.

The interactive prototype was testing using 3 Computer Science

Honours students. Participants were briefed on a short background

of the app and asked to use the system for 10 minutes. Again, the

think aloud technique was employed and participants informally

interviewed after.

The majority of the feedback given related to the aesthetics, con-

sistency and feedback provided by the application. Users suggested

that more color and icons should be added. In addition to this, the

way users navigate from the tutorial to the rest of the application

should be uniform and take place through the side menu like other

pages. Multiple participants noted that they would like feedback

once they have successfully changed a field on their profile, such as

their username, password and profile picture. These usability issues

were corrected in the software implementation of the application.

A digital prototype of the Label Tweets page can be seen in Figure

1b.

6https://proto.io/

7https://material.io/guidelines/

Subsequent to this, since all navigation issues had been identified,

an interface flow diagram was drafted to map the user interactions

from screen to screen. This ensured that the navigation would be

implemented in accordance with user feedback from this iteration

and iteration 2. The interface flow diagram can be found in the

online appendix8.

4 SOFTWARE IMPLEMENTATION4.1 Development MethodologyAn Agile Software Development methodology [10] was adopted

throughout the course of development. This methodology was se-

lected as the project had a rigid timeline and needed to respond

to changing requirements from users, both characteristics of Agile

conducive projects [10]. Although a specific framework such as

Scrum or Kanban was not used, the 4 values common to all Agile

frameworks, as specified in the Agile Manifesto [6] were observed.

In addition to this, those principles shared between agile methodolo-

gies, namely: iterative and incremental development were followed[10]. How the development of the software artifact adhered to these

principles and the values in the Agile Manifesto, will be discussed

here.

4.1.1 Individuals and interactions more than processes and tools.Weekly meetings were held with both the project team and supervi-

sors. These meetings primarily focused on monitoring the progress

of development and outlined the plan for what would be developed

next.

8https://people.cs.uct.ac.za/~knmkri002/

4

4.1.2 Working software more than comprehensive documentation.Software was tested daily through the use of print statements, thus

ensuring that at given time, the software artifact was functional.

Documentation of the artifact was produced in the form of inline

code commenting as well as docstrings for all methods of every

class, following TypeScript convention of specifying the expected

behavior, parameters and return values9.

4.1.3 Customer collaboration more than contract negotiation.Users were involved throughout the design of the application and

their feedback given in each iteration incorporated for subsequent

design cycles (see Section 3). Furthermore, the majority of errors

identified by users through feedback from the final heuristic evalu-

ation were corrected (see Section 6.3)

4.1.4 Responding to change more than following a plan. An ini-

tial Gantt chart was created at the onset of project and was up-

dated in response to the progress of development. UML design

documentation was produced, however, focused only on the core

functionality of the application and most difficult features to de-

velop. Each diagram produced is discussed throughout the paper in

their corresponding phases of design and development.

4.1.5 Iterative and incremental development. An initial mini-

mum viable product was produced during iteration 1 (see Section

3.1.2) and built on incrementally throughout the development pro-

cess. Development was split into 6 small iterations corresponding

to the application features. Each feature is discussed in the order of

development in Section 4.3.

4.1.6 Development Environment and Tools. The application was

built using the WebStorm IDE10. This was selected since the IDE

is specifically designed for Web based technologies and provides

TypeScript support. The device was tested using both Chrome

Developer Tools11

and the Ionic Lab12, thus allowing the app to be

emulated, live reloaded in response to code changes, and debugged

in real-time.

4.1.7 Application Aesthetics. The application follows Google

Material Design throughout. The Ionic framework was selected

for development as components are prestyled in adherence to the

Material Design guidelines.

4.2 System ArchitectureThe crowdsourcing application was built using the Ionic/Angular

hybrid development framework, and thus is written primarily using

Web based technologies. The UI is then rendered in a WebView,

a native container for housing the app in a window. It further

employs additional plugins to access the device’s native capabilities.

An overview of the systems architecture can be seen in Figure 2.

4.2.1 Pages. Each page consists of 3 elements: an Ionic Tem-

plate, a stylesheet and an Angular component. The user interface is

represented by the Ionic template and is marked up in HTML, thus

defining the structure and postioning of items on each page. Ele-

ments on the page are aesthetically modified through the stylesheet

9http://typedoc.org/guides/doccomments/

10jetbrains.com/webstorm/

11developer.chrome.com/devtools

12http://blog.ionic.io/ionic-lab/

Figure 2: System Overview

and are edited using ScSS, a superset of CSS3. In contrast to this,

Ionic applications are built on the Angular Framework, and house

application logic in Angular components, which are written in

Typescript (a statically typed superset of JavaScript). Components

and Templates are connected via two way data binding. These in-

clude interpolation, where a value in a component can be passed to

an HTML element. For example, in a div, <div>{{name}}</div>, the

content of the div is bound to the name variable in the component.

Similarly, data flows from template to component through event-

binding, whereby a particular action can be bound to a function

that would execute application logic in the component.

4.2.2 Cordova Plugins. As the application executes in the We-

bView, access to Android native functionality is required. These

features employed in the crowdsourcing app include the camera,

toasts (a pop-up similar to a tooltip) and alerts, and are accessed

through Cordova API bindings.

4.2.3 Firebase. The crowdsourcing application relies on client

side complexity therefore employs no server-side code. However,

it still makes use of Firebase, a platform agnostic backend-as-a-

service technology. The application accesses various Firebase ser-

vices through their JavaScript APIs. Firebase Authentication is em-

ployed to provide email/password authentication and user man-

agement. User, tweet and annotation data is stored and synced in

realtime using the cloud-hosted, noSQL database. An example of

database nodes can be seen in Figure 3. Furthermore, all images are

stored, managed, and retrieved through Firebase’s Cloud Storage.

4.2.4 Ng2-tweet module. All tweets are displays in theWebView

using the Ng2-tweet module, an open source Angular2 package,

that renders Twitter’s tweet widget. Through interpolation (see

section 4.2.1) the tweetID property is bound to the id supplied. Themodule then scans the DOM and dynamically inserts the rendered

tweet into the page at runtime.

5

(a) Tweet Node

(b) User Node

Figure 3: Database Nodes

4.3 Application Features4.3.1 User Authentication and Login. The application employs

the Firebase authentication service to register, login and manage

users through their Javascript authentication API. Email/ password

authentication is used throughout the application. New users are

registered through the createUserWithEmailAndPassword() method,

and the generated UUID is returned. The UUID is then added and

referenced in the Firebase Database so that additional information

such as which tweets as well as how many a user has annotated can

be stored with their user profile. Once a user has been registered,

they are able to log in via the signInWithEmailAndPasswordMethod()and logout through the signOut() method. In the event that a user

forgets their password, an automated reset password email can be

sent to the users through the sendPasswordResetEmail() method.

4.3.2 Annotation of Tweets. Annotation of tweets is the core

function of the application and most complex use case. As such,

the user interaction with the system and all function calls between

classes were mapped out prior to development through a system

sequence diagram. The diagram can be found in the online appen-

dix13.

Through the annotation page, tweets are presented to users

one at a time for labelling. For each tweet, a database snapshot

of all related data is pulled from the Firebase Database using the

database.ref.once() method. The tweet is rendered in the UI through

a call to the Ng2-tweet module together with 3 annotation options:

REAL, FAKE and CANNOT DECIDE (See Figure 1c).

Since multiple users on different devices can simultaneously an-

notate the same tweet, consideration has been given to concurrent

updates to the database. When a tweet is annotated, the response

is stored with the current tweet node in the database (See Figure

3a) using the database.ref.transaction() method, thus ensuring the

13https://people.cs.uct.ac.za/~knmkri002/

data is updated atomically and no updates are lost via simultaneous

writes. The response is further recorded with the user profile node

(See Figure 3b), such that all annotations can be associated with the

specific labeller.

Users are further given the option to change their opinion through

the UNDO button. Every tweet annotated per session is stored in a

stack. On pressing undo, the annotation is popped from the stack

and the transaction reversed in the database.

4.3.3 Profile Management. From the profile page, users are able

to view and edit their personal information such as their pass-

word, username and email. Personal details fields are automati-

cally pulled and populated from the Firebase database from fire-base.database().ref(userProfile/uuid) node. By tapping a field, users

are directed to a Modal Page that allows editing of personal infor-

mation. Feedback is provided in the form of toasts, if changes are

successfully propagated to the database, and alerts, if errors occur

during the process (See Section 4.3.5).

Users are able to view and and change their profile picture di-

rectly from the Profile Page. Through the Cordova Camera Plu-

gin14

users can access their gallery and select a new profile picture

via the cameraPlugin.getPicture() function. A Base64 encode im-

age is returned and then propagated to Firebase Cloud Storage

through the firebase.storage().ref(/userProfilePic/imageName) .put-String() method.

4.3.4 Leaderboard. From the leaderboard, users are able to com-

pare their annotation contribution to other users. A database snap-

shot of users, ordered by the number of tweets they have anno-

tated, is fetched from the database using the userProfilesRef .order-ByChild(’noOfAnnotatedTweets’).once() method call and rendered

in the user interface. Their rank, username and annotation con-

tribution are displayed together. In addition to this, each user’s

profile picture is retrieved from the Firebase Cloud Storage from

the picture URL that is stored with each profile node in the database.

A user is able to easily differentiate their place on the leaderboard

from other users. This is achieved by changing the glow around

their profile picture and annotation contribution to red, and having

their rank and username indicated in bold.

4.3.5 Tutorial and FAQs. To improve learnability of the sys-

tem, tutorial and frequently asked questions (FAQs) pages were

added. Both pages can be accessed directly from the side menu.

The questions for the FAQs page were derived from the feedback

given during each prototype iteration as well the from the final

heuristic evaluation. Although users can access the tutorial from

the side menu, they are automatically directed to the tutorial page

upon signing up. The tutorial covers the main functions of the

application, including how to label tweets, how to manage your

profile, and how to understand the information presented on the

leaderboard.

4.3.6 System Feedback. System feedback is provided to the user

in three forms: pop-up alerts, toasts and underlined form validators.

Alerts are presented to users to confirm changes made to profile

information as well as logout confirmation. Likewise, toasts are

14https://github.com/cordova-plugin-camera-preview/

cordova-plugin-camera-preview

6

used as feedback subsequent to changing profile information and

annotation of a tweet. Underlined form validators are displayed

to users when the the input supplied does not match a predefined

regular expression such as email regex or minimum length input.

Examples of feedback can be seen in Figure 4.

(a) Toast

(b) Alert

(c) Underlined Form Validator

Figure 4: Examples of System Feedback

5 CROWDSOURCING EVALUATION5.1 Gold Standard Data SetInitially the image-verification-corpus15, a data set of fake and real

tweets containing images, was to be used to assess whether crowd-

sourcing is a viable means of determining the truthfulness of tweets.

However, upon inspection, the tweets relate to events that South

African users may not have been exposed to or familiar with such

as the 2012 Hurricane Sandy and 2013 Boston Marathon bombing.

Hence, this necessitated collecting and labelling a data set for

use in this study. This section describes this process in three steps

(i) identification of fake accounts, (ii) identification of real accounts,

and (iii) labelling of tweets as fake and real.

5.1.1 Identification of Fake Accounts. Initially 12 fake Twitter

accounts had been identified by South African online news web-

sites16 17 18 19 20

. Furthermore, 12 self declared spoof accounts

were found and one fake account21

where a verified account of

the user already existed on Twitter. In total, 25 fake accounts were

identified.

5.1.2 Identification of Legitimate Accounts. Twitter provides amechanism to authenticate accounts that are considered to be in

15github.com/MKLab-ITI/image-verification-corpus

16city-press.news24.com/Trending/5-of-sas-best-twitter-parody-accounts-20151226

17www.fin24.com/Tech/News/SAs-most-controversial-fake-Twitter-accounts-20140709

18www.thedailyvox.co.za/woolworths-parody-account-inflames-twitter-users/

19www.enca.com/south-africa/sa-news-organisations-targeted-in-fake-twitter-accounts

20techfinancials.co.za/2017/05/24/mtn-distances-parody-twitter-account/

21twitter.com/maimane_mmusi

the public interest22. A blue tick badge is placed next to a user’s

username to indicate that they have been verified. Accounts are

verified manually and on a per request basis by Twitter23.

Using verified accounts as a proxy for real accounts, 23 real ac-

counts were identified. Furthermore, an additional two accounts

were identified as real by extracting the usernames directly from

official websites of news sites24

and political parties25. In total 25

South African accounts related to the following domains: govern-

ment, politics, journalism, and media were identified as real.

(a) Real Tweet

(b) Fake Tweet

Figure 5: Examples of Fake and Real Tweets

5.1.3 Labelling of Fake and Real Tweets. Using the heuristic thatfake accounts would attempt to portray the views of a legitimate

user in a deceptive way26, all tweets from the identified fake ac-

counts would be labelled as fake. Conversely, real accounts would

express their own views. Hence, tweets from the identified real

accounts are labelled real. Collectively, the gold standard data set

consists of 50 tweets, 25 fake and 25 real.

5.2 Crowdsourced Data SetIn order to obtain crowdsourced labels for the tweets identified in

Section 5.1, 14 users were recruited to annotate tweets through the

mobile application. Users were selected on the basis that they were

South African and active social media users, thus ensuring that

they were familiar with the South African context of the tweets.

In accordance with Twitter’s Impersonation Policy27

that states,

“Twitter accounts portraying another person ina confusing or deceptive manner may be perma-nently suspended under the Twitter imperson-ation policy”,

22support.twitter.com/articles/119135

23support.twitter.com/articles/20174631

24www.heraldlive.co.za/

25udm.org.za/bantu-holomisa/

26https://support.twitter.com/articles/18366

27support.twitter.com/articles/18366

7

Table 1: Summary of Labelled Data from Participants

Real Fake Can’t Decide Total

p1 23 21 6 50

p2 21 20 9 50

p3 20 22 8 50

p4 28 17 5 50

p5 25 22 3 50

p6 27 21 2 50

p7 26 24 0 50

p8 29 21 0 50

p9 28 20 2 50

p10 32 16 2 50

p11 32 18 0 50

p12 25 16 9 50

p13 25 20 5 50

p14 16 7 27 50

Crowdsourced 27 23 0 50

Total 384 288 78 750

users were instructed that any tweet that portrays another usersin a confusing or deceptive manner should be considered Fake.Participants were further directed that they should not use any

external resources in the evaluation and that they could label each

tweet as either Real, Fake or Cannot Decide. Figure 1c gives an

example of the interface presented to users for annotation.

Participants were given 1 hour to annotate 50 tweets identified

in Section 5.1. All tweets were presented to them in a randomized

order through the mobile application.

Subsequent to this, a crowdsourced label for each tweet was

determined. In order to assign the crowdsourcing label, the option

that was selected by the majority was taken. For example if the

response recorded for a tweet was 8 Real, 4 Fake and 2 CannotDecide, then the crowdsourced label given would be Real. If thereis no response in the majority, the crowdsourced labelled assigned

would be Cannot Decide. A summary of the data collected can be

found in Table 1.

5.3 ResultsTo determine the success of crowdsourcing as a means of labelling

the truthfulness of tweets, the percentage of correctly classified

instances identified through crowdsourcing is compared with that

of each participant as well as the performance of the average par-

ticipant.

For each tweet where the label assigned by a participant corre-

sponds to the label of the gold standard data, it is taken that the

participant correctly classified the tweet instance. For example, if

the participant’s label was fake and the gold standard label was

fake, the instance was correctly classified. Conversely, where the

label assigned by the participant does not correspond to the gold

standard data label, or where the label assigned by the participant

is cannot decide, it is taken that the tweet instance was incorrectly

classified. For example, if the participant’s label was real and the

gold standard label was fake, the instance was incorrectly classified.This method of determining correctly classified instances, applies

as well to comparing the crowdsourcing labels to the gold standard

labels. Tables 2 and 3 provide a summary of the results.

Crowdsourcing achieved a high level of accuracy at both iden-

tifying real and fake tweets, classifying real tweets correctly with

96% accuracy and fake tweets with 88% accuracy. Crowdsourcing

was outperformed by only one annotator, p12, at identifying realtweet instances with p12 correctly classifying 100% of real tweets

correctly. Crowdsourcing outperformed every annotator at identi-

fying fake tweet instances, with the next best annotators, p1 andp3, correctly identifying 84% of fake tweets. In contrast, crowd-

sourcing convincingly outperformed the average annotator in both

categories. By achieving 13.14% greater accuracy at identifying real

tweet instances, and 21.15% greater accuracy at identifying fake

tweet instances, thus suggesting that crowdsourcing is better at

annotating the truthfulness of tweets than the average annotator.

Table 2: Correctly Identified Real Tweets

No. Correct % Correct

p1 22 88

p2 15 60

p3 20 80

p4 23 92

p5 17 68

p6 23 92

p7 18 72

p8 23 92

p9 23 92

p10 23 92

p11 24 96

p12 25 100

p13 22 88

p14 12 48

Average 20,71 82,86

Crowdsourced 24 96

6 USABILITY EVALUATION6.1 Users and Experiment ProcessTo identify any usability issues of the the mobile application, a

heuristic evaluation was administered.

The experiment was conducted in 1 hour long sessions, in the

Computer Science Honours labs. Four Computer Science Honours

students were recruited to evaluate the system. All students had

taken the Human Computer Interaction Honours module and hence

would be familiar with Nielsen’s Heuristics28.

Users were given a brief background of the application and asked

to perform a series of tasks to familiarize themselves with the

system. They were asked to try to complete the tasks unassisted,

however, if they ran into critical errors that could not be recovered

from they could ask the facilitator for guidance. Subsequent to this,

users were given a list of all 10 of Nielsen’s Heuristics and asked

to explain every problem where elements of the interface violated

the heuristic. Users were then instructed to rate the severity of

28tfa.stanford.edu/download/TenUsabilityHeuristics.pdf

8

Table 3: Correctly Identified Fake Tweets

No. Correct % Correct

p1 21 84

p2 16 64

p3 21 84

p4 16 64

p5 16 64

p6 19 76

p7 17 68

p8 19 76

p9 19 76

p10 14 56

p11 17 68

p12 16 64

p13 18 72

p14 5 20

Average 16,71 66,85

Crowdsourced 22 88

each problem as either critical, high, medium or low based on the

following definitons29:

• Critical: Seriously impedes usability and function. There

is no workaround to these problems

• High: A serious condition that impairs the operation of

one or more user tasks that cannot be easily circumvented

or avoided.

• Medium: A non critical problem that causes moderate con-

fusion or irritation

• Low: Non-critical problems or general questions about the

task process. These are minor inconsistencies that cause

hesitation

In total, 26 unique usability issues were identified across all

heuristic categories. These include 1 critical, 0 high, 2 moderate, 9low and 12 unspecified severity problems.

6.2 Usability Problems6.2.1 Visibility of System Status. Primarily positive feedback

was given by participants. Users indicated that feedback was clearly

visible from the login page when entering form fields and that feed-

back given in response to labelling of tweets were understandable.

Furthermore, users appreciated that feedback was immediately

shown after every action and that messages were simple and easy

to understand. However, a point of moderate irritation identified is

that users were unable to view password fields in plaintext.

6.2.2 Match Between System and Real World. Some feedback

messages identified used system orientated and technical terms, for

example: informing users of their labelling contribution with the

phrase "Number of tweets annotated" was unclear and difficult to

understand. In contrast to this, icons used were found to be easy

to recognize and elements such as buttons, fields and menu items

easily identifiable.

29https://msu.edu/~bowsersa/HeuristicEvaluationForm.pdf

6.2.3 User Control and Freedom. One critical error was encoun-tered by multiple users, with toast messages occasionally failing to

dismiss. Users were unable to recover from the error as the toast

concealed the side menu, thus preventing users from navigating

away from the page. Likewise, users were unable to fix a tweet

label after accidentally already marking it, e.g. pressing fake ratherthan real. This was likely to occur as buttons were not spaced far

enough apart. The system further displayed unintended behavior,

whereby clicking on the tweet itself would automatically open the

tweet in the Twitter app on the current user’s account.

6.2.4 Consistency and Standards. 4 errors were identified in

this category. Users are accustomed to being presented with an

email keyboard when entering an email text field on other systems.

However, through the application they did not have access to the

@ or .com keys.

There was a further mismatch between the way fields are in-

putted through the login and profile pages. The login pages make

use of input fields directly on the screenwhereas users are presented

with an alert through the profile page.

Confusion was caused as the tutorial failed to adequately ad-

dress how a user’s score is calculated, and whether or not labelling a

tweet as "CANNOT DECIDE" contributed to their total. The tutorial

presented further confusion, since it contained outdated informa-

tion of how to label tweets, with users instructed to label tweets

on a 5 point scale as opposed to TRUE or FALSE in the most up to

date version of the application.

6.2.5 Error Prevention. User were able to edit their password,

username and email in free form, without any form validation. If

they subsequently made a mistake they would be presented with

a generic error message. In the same vein, users were not given

the confirmation options when making these edits or attempting

to logout.

6.2.6 Recognition Rather than Recall. In general positive feed-

back was given with users indicating that no prior information

was required to perform actions and that icons on pages made it

easier to navigate the system. However, users did communicate

that the tutorial elements were too small to recognize elements on

the screen.

6.2.7 Flexibility and Efficiency of Use. Multiple users could not

identify shortcuts in the application, but indicated that the app was

easy to use, that buttons to label tweets were easily accessible at

the bottom of the page and that the app was - "very slick".

6.2.8 Aesthetic and Minimalist Design. Participants gave mixed

feedback with some stating that all dialogues show the correct

amount of necessary information, with others claiming that select

error messages contain too much text.

6.2.9 Help Users Recognize, Diagnose and Recover from Errors.Select users observed that not all error messages were expressed in

simple language and as a result were unclear. Error messages could

be further emphasized by highlighting them in red.

6.2.10 Help andDocumentation. Although the systemwas found

to be intuitive, the tutorial was appreciated to assist with learning.

9

It was expressed that a frequently asked questions page be provided

to supplement the current documentation.

6.3 Error CorrectionOf the 26 unique errors that were identified, 25 were rectified and 1

left unresolved. Prominent issues were fixed first, with disabling the

clickable area of a tweet, thus preventing the app from redirecting

to Twitter. Furthermore, toasts that refuse to dismiss were corrected

by upgrading the ionic-angular package to version 3.5.2 and adding

a close button to each toast thus allowing users to recover in the

event that toast persists. In addition to this, undo functionality was

added so that users could edit mistakes made during labelling.

Five errors were eliminated by adding a Frequently Asked Ques-

tions page to improve documentation and specifically address the

questions "How is my score calculated on the leaderboard?", "does

labelling a tweet as CANNOT DECIDE add points to my score?"

and "why is my score not calculated based on how many tweets I

labelled correctly?".

Email keyboard and show password errors were corrected by

toggling the type property of the input fields and 3 additional errors

were solved by making minor edits to the stylesheets. In addition

to this, the language used throughout the application and error

messages was simplified.

The only issue left unresolved was the lack of app shortcuts.

The decision was taken not add any as the application received

generally positive feedback for its simplicity and therefore does not

require shortcuts due to its small size.

7 LIMITATIONSThe identified requirement that users should be able to annotate

tweets on a credibility scale could not be met. This is due to the fact

that the data set used to evaluate the performance of crowdsourcing

had only ’TRUE’ and ’FALSE’ labels. As such, users are presented

with a binary scale in the application.

A limitation of the research is that participants selected to an-

notated tweets, were merely active social media users and not

necessarily Twitter users. As a result, not all users would have been

familiar with the structure of tweets and the indicators of credibility

of tweets. Active Twitter users may have assigned different truth

labels.

8 ETHICAL, PROFESSIONAL AND LEGALISSUES

Ethical clearance was given by UCT’s Faculty of Science Research

Ethics Committee. Furthermore, clearance was granted by the Di-

rector of Student Affairs since UCT students would be used in the

study. The ethical clearance certificate can be obtained with the

following certificate code: FSREC 50 —2017. Throughout the studyusers were asked to sign an informed voluntary consent form. They

were briefed that they could pull out the study at any point in time,

that the study posed no harmful risks, and that their anonymity

would be maintained throughout the study.

The software artifact produced conforms to both the Twitter

Developer Policy30and Firebase Terms of Service

31. Furthermore,

30developer.twitter.com/en/developer-terms/policy

31firebase.google.com/terms/

the crowdsourcing application is Open Source and will be released

under the Creative Commons license, in congruence with UCT’s IP

policy32.

9 CONCLUSIONSIn this paper, we presented the design and implementation of a mo-

bile crowdsourcing application to annotate the veracity of tweets.

A user centered design approach was adopted and carried out over

successive iterations. Through the design process, individual inter-

views, focus groups, and paper and digital prototyping phases were

conducted. This culminated in a hybrid mobile application being

built using Ionic, Cordova, Angular and Firebase technologies.

The results from the final heuristic evaluation gave generally

positive feedback, however, it revealed 26 usability issues. These

include 1 critical, 0 high, 2moderate, 9 low and 12 unspecified severityproblems. All usability errors except one have since been corrected.

We further evaluated whether mobile crowdsourcing can be

used as a viable means of annotating the truthfulness of tweets.

The study conducted with 14 South African active social media

users showed that crowdsourcing performed better than every

participant, bar one, at identifying real tweets, achieving 96% accu-

racy. Furthermore, crowdsourcing outperformed every annotator atidentifying fake tweets, correctly classifying 88% of fake tweet in-

stances. In addition to this, the crowdsourced annotations achieved

13.14% greater accuracy at identifying real tweet instances, and

21.15% greater accuracy at identifying fake tweet instances com-

pared to the average accuracy in each category. Thus, suggesting

that crowdsourcing is better at annotating the veracity of tweets

than the average annotator, affirming that mobile crowdsourcing

can be used as a viable means of annotating the truthfulness of

tweets.

10 FUTUREWORKSince the code base has beenwritten withmodularity and extensibil-

ity in mind, an interesting avenue to explore would be to generalize

the application to allow for crowdsourcing the credibility of other

social media data such as Facebook and Instagram posts. Likewise,

the application could be further generalized to crowdsource not

only credibility, but crowdsource data required for systems that

utilize machine learning, such as sentiment analysis, translation,

and handwriting recognition.

The application can be extended to be deployed on other mobile

platforms. This will require minimal additional work, as having

being built with Cordova and Ionic, the hybrid app can be deployed

easily on both IOS and Windows with the code base not having to

be rewritten.

11 ACKNOWLEDGEMENTSI would like to thank Mr Selvas Mwanza and Dr Hussein Sule-

man for their continual guidance and invaluable input throughout

the development of this project. Furthermore, I wish to express

my sincere gratitude to other members of the SASITWIT team:

Kristin Kinmont and Michelle Lu. Without their encouragement

and support this work would not have been completed.

32www.uct.ac.za/downloads/uct.ac.za/about/policies/intellect_property.pdf

10

REFERENCES[1] Aggarwal, A., Rajadesingan, A., and Kumaraguru, P. Phishari: Automatic

realtime phishing detection on twitter. In eCrime Researchers Summit (eCrime)(2012), IEEE, pp. 1–12.

[2] Ali, K., Al-Yaseen, D., Ejaz, A., Javed, T., and Hassanein, H. S. Crowdits:

Crowdsourcing in intelligent transportation systems. InWireless Communicationsand Networking Conference (WCNC), 2012 IEEE (2012), IEEE, pp. 3307–3311.

[3] Alt, F., Shirazi, A. S., Schmidt, A., Kramer, U., and Nawaz, Z. Location-based

crowdsourcing: extending crowdsourcing to the real world. In Proceedings ofthe 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries(2010), ACM, pp. 13–22.

[4] Alvaro, N., Conway, M., Doan, S., Lofi, C., Overington, J., and Collier,

N. Crowdsourcing twitter annotations to identify first-hand experiences of

prescription drug use. Journal of biomedical informatics 58 (2015), 280–287.[5] Arslan, M. Y., Singh, I., Singh, S., Madhyastha, H. V., Sundaresan, K., and

Krishnamurthy, S. V. Computing while charging: Building a distributed com-

puting infrastructure using smartphones. In Proceedings of the 8th internationalconference on Emerging networking experiments and technologies (2012), ACM,

pp. 193–204.

[6] Beck, K., Beedle, M., Van Bennekum, A., Cockburn, A., Cunningham, W.,

Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., et al.Manifesto

for agile software development.

[7] Bessho, F., Harada, T., and Kuniyoshi, Y. Dialog system using real-time

crowdsourcing and twitter large-scale corpus. In Proceedings of the 13th AnnualMeeting of the Special Interest Group on Discourse and Dialogue (2012), Associationfor Computational Linguistics, pp. 227–231.

[8] Brabham, D. C. Crowdsourcing as a model for problem solving: An introduction

and cases. Convergence 14, 1 (2008), 75–90.[9] Castillo, C., Mendoza, M., and Poblete, B. Information credibility on twitter.

In Proceedings of the 20th International Conference on World Wide Web (New York,

NY, USA, 2011), WWW ’11, ACM, pp. 675–684.

[10] Cohen, D., Lindvall, M., and Costa, P. Agile software development. DACSSOAR Report 11 (2003).

[11] Dong, Y. F., Kanhere, S., Chou, C. T., and Bulusu, N. Automatic collection of

fuel prices from a network of mobile cameras. Lecture Notes in Computer Science5067 (2008), 140–156.

[12] Eagle, N. txteagle: Mobile crowdsourcing. Internationalization, design and globaldevelopment (2009), 447–456.

[13] Gonçalves, B., and Sánchez, D. Crowdsourcing dialect characterization

through twitter. PloS one 9, 11 (2014), e112074.[14] Grier, C., Thomas, K., Paxson, V., and Zhang, M. @spam: The underground on

140 characters or less. In Proceedings of the 17th ACM Conference on Computer andCommunications Security (New York, NY, USA, 2010), CCS ’10, ACM, pp. 27–37.

[15] Gupta, A., and Kumaraguru, P. Credibility ranking of tweets during high

impact events. In Proceedings of the 1st Workshop on Privacy and Security inOnline Social Media (New York, NY, USA, 2012), PSOSM ’12, ACM, pp. 2:2–2:8.

[16] Gupta, A., Kumaraguru, P., Castillo, C., and Meier, P. Tweetcred: Real-time

credibility assessment of content on twitter. In International Conference on SocialInformatics (2014), Springer, pp. 228–243.

[17] Gupta, A., Lamba, H., Kumaraguru, P., and Joshi, A. Faking sandy: Char-

acterizing and identifying fake images on twitter during hurricane sandy. In

Proceedings of the 22nd International Conference on World Wide Web (New York,

NY, USA, 2013), WWW ’13 Companion, ACM, pp. 729–736.

[18] Gupta, A., Thies, W., Cutrell, E., and Balakrishnan, R. mclerk: enabling

mobile crowdsourcing in developing regions. In Proceedings of the SIGCHIConference on Human Factors in Computing Systems (2012), ACM, pp. 1843–1852.

[19] Kwak, H., Lee, C., Park, H., and Moon, S. What is twitter, a social network or a

news media? In Proceedings of the 19th international conference on World WideWeb (2010), ACM, pp. 591–600.

[20] Lee, S., and Kim, J. Warningbird: Detecting suspicious urls in twitter stream. In

NDSS (2012), vol. 12, pp. 1–13.[21] Liu, Y., Lehdonvirta, V., Alexandrova, T., Liu, M., and Nakajima, T. Engaging

social medias: case mobile crowdsourcing.

[22] Mendoza, M., Poblete, B., and Castillo, C. Twitter under crisis: Can we trust

what we rt? In Proceedings of the First Workshop on Social Media Analytics (NewYork, NY, USA, 2010), SOMA ’10, ACM, pp. 71–79.

[23] Mohan, P., Padmanabhan, V. N., and Ramjee, R. Nericell: rich monitoring of

road and traffic conditions using mobile smartphones. In Proceedings of the 6thACM conference on Embedded network sensor systems (2008), ACM, pp. 323–336.

[24] Morris, M. R., Counts, S., Roseway, A., Hoff, A., and Schwarz, J. Tweeting is

believing?: Understanding microblog credibility perceptions. In Proceedings ofthe ACM 2012 Conference on Computer Supported Cooperative Work (New York,

NY, USA, 2012), CSCW ’12, ACM, pp. 441–450.

[25] Narula, P., Gutheim, P., Rolnitzky, D., Kulkarni, A., and Hartmann, B.

Mobileworks: A mobile crowdsourcing platform for workers at the bottom of

the pyramid. Human Computation 11 (2011), 11.[26] O’Donovan, J., Kang, B., Meyer, G., Hollerer, T., and Adalii, S. Credibility

in context: An analysis of feature distributions in twitter. In Proceedings of the2012 ASE/IEEE International Conference on Social Computing and 2012 ASE/IEEEInternational Conference on Privacy, Security, Risk and Trust (Washington, DC,

USA, 2012), SOCIALCOM-PASSAT ’12, IEEE Computer Society, pp. 293–301.

[27] Okolloh, O. Ushahidi, or âĂŸtestimonyâĂŹ: Web 2.0 tools for crowdsourcing

crisis information. Participatory learning and action 59, 1 (2009), 65–70.[28] Phuttharak, J., and Loke, S. W. Logiccrowd: A declarative programming

platform for mobile crowdsourcing. In Trust, Security and Privacy in Computingand Communications (TrustCom), 2013 12th IEEE International Conference on(2013), IEEE, pp. 1323–1330.

[29] Prpić, J., Shukla, P. P., Kietzmann, J. H., and McCarthy, I. P. How to work a

crowd: Developing crowd capital through crowdsourcing. Business Horizons 58,1 (2015), 77–85.

[30] Ratkiewicz, J., Conover, M., Meiss, M. R., Gonçalves, B., Flammini, A., and

Menczer, F. Detecting and tracking political abuse in social media. ICWSM 11(2011), 297–304.

[31] Schenk, E., and Guittard, C. Crowdsourcing: What can be outsourced to the

crowd, and why. In Workshop on Open Source Innovation, Strasbourg, France(2009), vol. 72.

[32] Starbird, K. Digital volunteerism during disaster: Crowdsourcing information

processing. In Conference on human factors in computing systems (2011), pp. 7–12.[33] Stevens, M., and DâĂŹHondt, E. Crowdsourcing of pollution data using

smartphones. InWorkshop on Ubiquitous Crowdsourcing (2010).

[34] Wang, A. H. Detecting spam bots in online social networking sites: A machine

learning approach. In IFIP Annual Conference on Data and Applications Securityand Privacy (2010), Springer, pp. 335–342.

[35] Wang, A. H. Don’t follow me: Spam detection in twitter. In Proceedings of the2010 International Conference on Security and Cryptography (SECRYPT) (2010),IEEE, pp. 1–10.

[36] Yadav, K., Kumaraguru, P., Goyal, A., Gupta, A., and Naik, V. Smsassassin:

Crowdsourcing drivenmobile-based system for sms spam filtering. In Proceedingsof the 12th Workshop on Mobile Computing Systems and Applications (2011), ACM,

pp. 1–6.

[37] Yan, T., Marzilli, M., Holmes, R., Ganesan, D., and Corner, M. mcrowd: a

platform for mobile crowdsourcing. In Proceedings of the 7th ACM Conference onEmbedded Networked Sensor Systems (2009), ACM, pp. 347–348.

[38] Yardi, S., Romero, D., and Schoenebeck, G. Detecting spam in a twitter network.

First Monday 15, 1 (2009).

11