u nderstanding w ikipedia niki kittur [email protected]
TRANSCRIPT
![Page 2: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/2.jpg)
Slowing growth
• Since 2007, slowing growth
Why?• Fewer new topics to
write about• Growing resistance to
new contributions
Proportion reverted edits (by editor class)
Number of active editors per month
Suh, Convertino, Chi, & Pirolli, 2009
![Page 3: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/3.jpg)
Wisdom of crowds poll
What proportion of Wikipedia (in words) is made up of
articles?
0-25% | 25-50% | 50-75% | 75-100%
![Page 4: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/4.jpg)
Wisdom of crowds poll
![Page 5: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/5.jpg)
Article
![Page 6: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/6.jpg)
Discussion
![Page 7: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/7.jpg)
Discussion
![Page 8: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/8.jpg)
Edit history
![Page 9: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/9.jpg)
Edit history
![Page 10: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/10.jpg)
Policies + Procedures
![Page 11: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/11.jpg)
How does it work?
• “Wisdom of crowds” - Many independent judgments– “with enough eyeballs all bugs are shallow”
• More contributors ->– more information– fewer errors– less bias
![Page 12: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/12.jpg)
Wilkinson & Huberman, 2007
• Examined featured articles vs. non-featured articles– Controlling for PageRank (i.e., popularity)
• Featured articles = more edits, more editors
• More work, more people => better outcomes
Edits Editors
![Page 13: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/13.jpg)
Difficulties with generalizing results
• Cross-sectional analysis– Reverse causation: articles which become
featured may subsequently attract more people
• Coarse quality metrics– Fewer than 2000 out of >2,000,000 articles
are featured• What about coordination?
![Page 14: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/14.jpg)
Coordination costs
• Increasing contributors incurs process losses (Boehm, 1981; Steiner, 1972)
• Diminishing returns with added people (Hill, 1982; Sheppard, 1993)
– Super-linear increase in communication pairs– Linear increase in added work
• In the extreme, costs may exceed benefits to quality (Brooks, 1975)
• The more you can support coordination, the more benefits from adding people“Adding manpower to a late
software project makes it later”
Brooks, 1975
![Page 15: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/15.jpg)
Research question
To what degree are editors in Wikipedia working independently
versus coordinating?
![Page 16: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/16.jpg)
Research infrastructure
• Analyzed entire history of Wikipedia– Every edit to every article
• Large dataset (as of 2008)– 10+ million pages– 200+ million revisions– 2.5+ Tb
• Used distributed processing– Hadoop distributed filesystem– Map/reduce to process data in parallel– Reduce time for analysis from weeks to
hours
![Page 17: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/17.jpg)
Types of work
Direct work Editing articles
Indirect workUser talk, creating
policy
Maintenance work Reverts, vandalism
![Page 18: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/18.jpg)
Less direct work
• Decrease in proportion of edits to article page
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
2001 2002 2003 2004 2005 2006
Edi
t pr
opor
tion
70%
![Page 19: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/19.jpg)
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
2001 2002 2003 2004 2005 2006
Ed
it P
rop
ort
ion
More indirect work
• Increase in proportion of edits to user talk
8%
![Page 20: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/20.jpg)
More indirect work
• Increase in proportion of edits to user talk
• Increase in proportion of edits to policy pages
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
2001 2002 2003 2004 2005 2006
Edi
t pr
opor
tion 11
%
![Page 21: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/21.jpg)
More maintenance work
• Increase in proportion of edits that are reverts
00.020.040.060.08
0.10.120.140.160.18
0.2
2001 2002 2003 2004 2005 2006
Ed
it p
rop
ort
ion
7%
![Page 22: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/22.jpg)
More wasted work
• Increase in proportion of edits that are reverts
• Increase in proportion of edits reverting vandalism
00.005
0.010.015
0.02
0.0250.03
2001 2002 2003 2004 2005
Ed
it p
rop
ort
ion
1-2%
![Page 23: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/23.jpg)
Global level
• Coordination costs are growing– Less direct work (articles)+ More indirect work (article talk, user,
procedure)+ More maintenance work (reverts, vandalism)
Kittur, Suh, Pendleton, & Chi, 2007
![Page 24: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/24.jpg)
Research question
How does coordination impact quality?
![Page 25: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/25.jpg)
Coordination types
• Explicit coordination– Direct communication among editors
planning and discussing article• Implicit coordination
– Division of labor and workgroup structure– Concentrating work in core group of editors
Leavitt, 1951; March & Simon, 1958; Malone, 1987; Rouse et al., 1992; Thompson, 1967
![Page 26: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/26.jpg)
Explicit coordination: “Music of Italy”
planning
![Page 27: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/27.jpg)
Explicit coordination: “Music of Italy”
coverage
![Page 28: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/28.jpg)
Explicit coordination: “Music of Italy”
readability
![Page 29: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/29.jpg)
Coordination types
• Explicit coordination– Direct communication among editors
planning and discussing article• Implicit coordination
– Division of labor and workgroup structure– Concentrating work in core group of editors
Leavitt, 1951; March & Simon, 1958; Malone, 1987; Rouse et al., 1992; Thompson, 1967
![Page 30: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/30.jpg)
Implicit coordination: “Music of Italy”
![Page 31: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/31.jpg)
Implicit coordination: “Music of Italy”
TUF-KAT: Set scope and structure
![Page 32: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/32.jpg)
Implicit coordination: “Music of Italy”
Filling in by many contributors
![Page 33: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/33.jpg)
Implicit coordination: “Music of Italy”
Restructuring by Jeffmatt
![Page 34: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/34.jpg)
Research question
• What factors lead to improved quality?– More contributors– Explicit coordination
• Number of communication edits
– Implicit coordination• Concentration among editors
![Page 35: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/35.jpg)
Measuring concentration
• If an article has 100 edits and 10 editors, it could have:– 10 editors making 10 edits each
![Page 36: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/36.jpg)
Measuring concentration
• If an article has 100 edits and 10 editors, it could have:– 10 editors making 10 edits each– 1 editor making 90 edits
![Page 37: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/37.jpg)
Measuring concentration
• If an article has 100 edits and 10 editors, it could have:– 10 editors making 10 edits each– 1 editor making 90 edits
• Measure concentration with Gini coefficient
![Page 38: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/38.jpg)
Measuring concentration
• If an article has 100 edits and 10 editors, it could have:– 10 editors making 10 edits each– 1 editor making 90 edits
• Measure concentration with Gini coefficient
Gini = 0
![Page 39: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/39.jpg)
Measuring concentration
• If an article has 100 edits and 10 editors, it could have:– 10 editors making 10 edits each– 1 editor making 90 edits
• Measure concentration with Gini coefficient
Gini = 0 Gini ~ 1
![Page 40: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/40.jpg)
Measuring quality
• Wikipedia 1.0 quality assessment scale – Over 900,000 assessments– 6 classes of quality, from “Stub” up to
“Featured”– Top 3 classes require increasingly rigorous
peer review• Validated community assessments with
non-expert judges (r = .54***)
![Page 41: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/41.jpg)
Analysis
![Page 42: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/42.jpg)
Analysis
![Page 43: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/43.jpg)
Analysis
![Page 44: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/44.jpg)
Editors + coordination
1. Editors no effect on quality2. Communication increase in quality3. Concentration increase in quality
![Page 45: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/45.jpg)
Communication x Editors
• Communication does not scale to the crowd– Effective with few editors– Ineffective with many editors
![Page 46: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/46.jpg)
Concentration x Editors
• Concentration enables effective harnessing of the crowd– High concentration: more editors increase quality– Low concentration: more editors reduce quality
![Page 47: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/47.jpg)
Summary
• Wikipedia includes large degree of coordination
• Adding more editors does not improve quality– Coordination between editors is critical
• Type of coordination is important– Communication does not scale to large
groups– Concentration does scale to large groups
![Page 48: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/48.jpg)
TOOLS FOR SOCIAL COLLABORATION
![Page 49: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/49.jpg)
Profits and perils of user-generated content
• Content in Wikipedia can be added or changed by anyone
• Because of this, has become one of the most important information resources on the web– Top 10 most popular websites (Alexa.com)– Millions of contributors
• Also causes problems– Conflict between contributors– Unknown trustworthiness
![Page 50: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/50.jpg)
Denning et al. (2005)
• Risks with using Wikipedia– Accuracy of content– Motives of editors– Expertise of editors– Stability of article– Coverage of topics– Quality of cited information
Insufficient information to evaluate trustworthiness
![Page 51: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/51.jpg)
History flow
![Page 52: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/52.jpg)
Details
![Page 53: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/53.jpg)
Vandalism
![Page 54: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/54.jpg)
Anonymous contribution
M$: many anonymous contributors
Brazil: few anonymous contributors
![Page 55: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/55.jpg)
Edit war
![Page 56: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/56.jpg)
Conflict at the user level
• How can we identify conflict between users?
Kittur et al., 2007; Suh et al. 2007; Brandes & Lerner, 2008
![Page 57: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/57.jpg)
Terry Schiavo
Mediators
Sympathetic to parents
Sympathetic to husband
Anonymous (vandals/spammers)
![Page 58: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/58.jpg)
Dokdo/Takeshima opinion groups
Group A
Group B Group C
Group D
![Page 59: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/59.jpg)
Ekstrand & Riedl, 2009
![Page 60: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/60.jpg)
Ekstrand & Riedl (2009)
![Page 61: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/61.jpg)
Ekstrand & Riedl (2009)
![Page 62: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/62.jpg)
Trust
• Numerous studies surface trust-relevant information– Editors [Adler & Alfaro, 2007; Dondio et al., 2006; Zeng
et al., 2006]
– Stability [Suh et al., 2008]
– Conflict [Kittur et al., 2007; Viegas et al., 2004]
• But how much impact can this have on user perceptions in a system which is inherently mutable?
![Page 63: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/63.jpg)
What would make you trust Wikipedia more?
Nothing
![Page 64: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/64.jpg)
What would make you trust Wikipedia more?
“Wikipedia, just by its nature, is impossible to trust completely. I don't think this can necessarily be changed.”
![Page 65: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/65.jpg)
Hypotheses
1. Visualization will impact perceptions of trust
2. Compared to baseline, visualization will impact trust both positively and negatively
3. Visualization should have most impact when high uncertainty about article• Low quality• High controversy
![Page 66: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/66.jpg)
Design
• 3 x 2 x 2 design
Abortion
George Bush
Volcano
Shark
Pro-life feminism
Scientology and celebrities
Disk defragmenter
Beeswax
Controversial
Uncontroversial
High quality
Low quality
Visualization
• High trust• Low trust• Baseline
(none)
![Page 67: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/67.jpg)
Method
• Users recruited via Amazon’s Mechanical Turk– 253 participants– 673 ratings– 7 cents per rating– Kittur, Chi, & Suh, CHI 2008: Crowdsourcing user
studies
![Page 68: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/68.jpg)
Example: High trust visualization
![Page 69: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/69.jpg)
Example: Low trust visualization
![Page 70: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/70.jpg)
Summary info: Editor
• % from anonymous users
![Page 71: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/71.jpg)
Summary info: Editor
• % from anonymous users
• Last change by anonymous or established user
![Page 72: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/72.jpg)
Summary info: Stability
• Stability of words
![Page 73: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/73.jpg)
Summary info: Stability
• Instability
![Page 74: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/74.jpg)
Summary info: Conflict
• Instability• Conflict
![Page 75: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/75.jpg)
Results
1. Significant effect of visualization– High > low, p < .001
2. Both positive and negative effects– High > baseline, p < .001– Low < baseline, p < .01
3. No effect of article uncertainty– No interaction of
visualization with either quality or controversy
– Robust across conditions
![Page 76: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/76.jpg)
Results
1. Significant effect of visualization– High > low, p < .001
2. Both positive and negative effects– High > baseline, p < .001– Low < baseline, p < .01
3. No effect of article uncertainty– No interaction of
visualization with either quality or controversy
– Robust across conditions
![Page 77: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/77.jpg)
Results
1. Significant effect of visualization– High > low, p < .001
2. Both positive and negative effects– High > baseline, p < .001– Low < baseline, p < .01
3. No effect of article uncertainty– No interaction of
visualization with either quality or controversy
– Robust across conditions
![Page 78: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/78.jpg)
![Page 79: U NDERSTANDING W IKIPEDIA Niki Kittur nkittur@cs.cmu.edu](https://reader036.vdocuments.site/reader036/viewer/2022062314/56649e7c5503460f94b7dbb2/html5/thumbnails/79.jpg)