insemtives tutorial iswc2011 - session3
DESCRIPTION
TRANSCRIPT
![Page 1: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/1.jpg)
Crowdsourcing the annotation of dynamic Web content at seekda
www.insemtives.eu 1
Elena Simperl, University of Innsbruck, AT Markus Rohde, University of Siegen, DE
ISWC 2010
![Page 2: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/2.jpg)
Overview
• Context • Prototyping • Participatory Design • User Challenges • Conclusions
www.insemtives.eu 2
![Page 3: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/3.jpg)
Context
• Web services portal – Crawls for and indexes Web Services on the Web – Currently more than 28,500 indexed and monitored
• Problems – Services are not annotated or described – Limited search results and possibilities – Web APIs need to be confirmed by users
• Goal – Obtain more annotations by involving users in the annotation
process – Validate existing annotations, if any – „Catch them & keep them“
www.insemtives.eu 3
![Page 4: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/4.jpg)
Design decisions • Different annotation methods exist
– Keywords/tags – Categories – Natural language descriptions – Lightweight/fully-fledged semantic web service descriptions (e.g.
WSMO/Light, OWL-S, etc.) – Avoid complicated and demanding annotations (limit to tags, categories and
NL descriptions) • Use lightweight RDF ontologies in the background (e.g. to ease the
search) • SWS annotations might be integrated in the future
– Most users are not familiar with SWS – Difficult to integrate within the search (diverse frameworks and variants) – May hamper performance & usability
www.insemtives.eu 4
![Page 5: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/5.jpg)
Prototype Creation
www.insemtives.eu 5
![Page 6: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/6.jpg)
Design Recommendations
www.insemtives.eu 6
![Page 7: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/7.jpg)
Participatory Design
• Involve (end) users in prototyping • Users = experts of use/practice • Needs assessment -> Requirements Analysis • Exploit users‘ expertise and creativity in design
processes • Integrate evaluation in design processes • Repeated prototyping cycles
www.insemtives.eu 7
![Page 8: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/8.jpg)
Online Participatory Design
• Seekda‘s users are – anonymous – distributed worldwide
• Online communication via website • Creating opportunities for online participation
– Establish appropriate OPD process design – Develop adequate OPD tool (= dashboard)
www.insemtives.eu 8
![Page 9: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/9.jpg)
Stakeholders‘ Benefits
www.insemtives.eu 9
Users‘ Benefit
• Design follows users‘
needs
• Implement own ideas
• Insights in technology
and development
Seekda‘s Benefit
• Getting direct input from
users/customers
• Focussing on central
user requirements
• Getting to know
users/customers
![Page 10: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/10.jpg)
OPD Stakeholders/ Process Roles
• Project owner – Initiation, Management, Coordination – Facilitation
• Research/ Observer – Expert as neutral consultant
• Technical committee – Developers/Designers and Users – Process Decisions
• User committee
www.insemtives.eu 10
![Page 11: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/11.jpg)
OPD Workshop - Procedure • In General:
1. Technical committee chooses number of features out of forum discussions
2. Features open for user voting 3. Feature selection, implementation 4. Collect user feedback
• Duration per cycle: 6 weeks
www.insemtives.eu 11
![Page 12: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/12.jpg)
OPD Workshop – Cycles •1st Cycle
• Beginning November: Start of workshop (dashboard, technical committee, introduction)
• 2 weeks later: Identification of 5 most important features/wishes • 1 week later: Selection of 2 most popular features • Beginning of Dec: Short tasks for users • Mid December: End of cycle, feedback analysis • OPD Dashboard Improvements
•2nd Cycle • Execution dates: January – March • Goals for this cycle
• Increase motivation • Increase activity of participants • Focus more on usability/design and incentives
• Changes • Tasks first • Split into smaller parts, sequentially • Explained through screencasts • Example: go to the portal, search for xyz, identify redundant elements, most
important, … • OPD Dashboard Improvements
www.insemtives.eu 12
![Page 13: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/13.jpg)
PD of the Dashboard
On-site PD workshop: Requirements for the PD dashboard
www.insemtives.eu 13
![Page 14: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/14.jpg)
OPD Description
www.insemtives.eu 14
Process description for participants + Video Instruction
![Page 15: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/15.jpg)
OPD Announcement
13 www.insemtives.eu 15
![Page 16: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/16.jpg)
OPD Introduction
13 www.insemtives.eu 16
![Page 17: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/17.jpg)
OPD Dashboard (2nd Cycle)
www.insemtives.eu 17
Improvements: awareness feature and weekly tasks for participants
![Page 18: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/18.jpg)
Feature Selection and Voting
www.insemtives.eu 18
![Page 19: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/19.jpg)
OPD Workshop - Results • Numbers
~ 250 votes ~ 160 forum posts 15-20 active users
• User Background
• Web Services experts • Developers • Random visitors
• Feedback/Implementation • 18 suggested features • 6 concrete features implemented (ongoing) • Several implemented usability/design improvements
• Conclusions & Next Steps (ongoing) • Introduce challenge procedures • Ask specifically about guided processes (wizards) • Integrate OPD workshop directly from the platform
www.insemtives.eu 19
![Page 20: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/20.jpg)
Evalution (I) • Six interviews (~60 min) with participants
– Experiences – General evaluation – Critique, improvements
• Limitations of written communication -> Multi-media • Performance problems • Positive: Video instruction • Improvement: Awareness features/ notifications
www.insemtives.eu 20
![Page 21: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/21.jpg)
Evaluation (II) • Central features and Usability have been
improved • High quality feedback from users • Improved planning of features/
implementation based on early discussion with users
• (Perceived) assistance/ support for developers/ designers
• „Yeah, I think it succeeded. We got a lot of contribution from people […] and it showed this kind of workshop can work. This kind of methods.“
www.insemtives.eu 21
![Page 22: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/22.jpg)
Challenge 1: Amazon Mechanical Turk
Goal: Initial annotations for new and undescribed APIs Tasks available
• Confirm document is related to a Web API (yes/no) • Provide/improve description • Provide and confirm (bootstrapped) tags/concepts • Provide and confirm (bootstrapped) service categories • Rate document quality
Parameters • Qualification test • Min. approval rate per worker • Approx. Time needed per task • Reward
www.insemtives.eu 22
![Page 23: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/23.jpg)
www.insemtives.eu 23
![Page 24: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/24.jpg)
www.insemtives.eu 24
![Page 25: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/25.jpg)
www.insemtives.eu 25
![Page 26: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/26.jpg)
www.insemtives.eu 26
![Page 27: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/27.jpg)
Challenge 1: MTurk
Simple Annotation Wizard Phase 1
1. Setup • Initial set of 70 Web APIs • No qualification needed • Approx. Time needed: 15 minutes • Reward: 0,10$ • Description + Screencast (Walkthrough)
2. Manual evaluation (seekda) • Main focus on description and Yes/No question • Determine whether qualification is needed for workers • Determine whether wizard is understandable (usability) • Determine whether review tasks are needed
www.insemtives.eu 27
![Page 28: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/28.jpg)
Challenge 1: Mturk Phase 1 Results
Total: 70 API documents, 23 distinct workers Initial Question (Document about Web API, Yes/No)
• 49 documents correctly annotated (70%) • 21 as Yes, this document is about a Web API • 28 as No, this document is not related to a Web API
Description, Category, Tags
• ~ 15 submissions including all answers (description, category, tags) • 4 very good and extensive submissions • 8 complete and usable submissions
Phase 1, Problems
• Spam (10% - 15%) • Only few added category and descriptions • Most workers did not add tags
www.insemtives.eu 28
![Page 29: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/29.jpg)
Mturk: Phase 2 Changes
• Completion time decreased to 10min • Reward increased to 20c • Key questions are mandatory (description, tags,
category) • More strict evaluation criteria
– e.g.: at least, 1 category, 2 tags and a meaningful description have to be provided.
• Submitted a batch of 100
![Page 30: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/30.jpg)
MTurk: Phase 2 Results • 27 users (only 1 from the previous batch!) • Completion Times
– Min: 10 sec – Max: 9 min
• 10 wrong classifications – 5 of them are web pages with high quality annotations
• For correct classifications: – Mostly annotated with 2 tags – Top level category identification accurate in most cases – Mostly meaningful descriptions – Over 80% are accurate/satisfying
![Page 31: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/31.jpg)
MTurk: Phase 2 Results •Large number of confirmed Web APIs
–Feed back to crawler/analysis framework –Improving the initial index quality
• Large number of categorised/tagged Services/APIs –Feed back to bootstrapping Service –Improved search/navigation
• Detailed service descriptions for many Services at once
–Improved user experience –Improved user experience/satisfaction –Attract/motivate new visitors to participate
![Page 32: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/32.jpg)
Challenge 2: Mashups • Overall Goal
– Create and add mashup(s) using services / Web APIs listed on the seekda portal – Annotate used Services and APIs
• Timeline – Duration: 4 weeks
• Introduction of Task through: – Step by step guidelines – Set of rules – Example walkthrough
• Reward – Gadget (Samsung Galaxy S)
www.insemtives.eu 32
![Page 33: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/33.jpg)
Challenge 3: Long-Term Competition
• Provide annotations – become a top contributor • Collect Points
• Changes and/or improvements to annotations • New annotations • Weighting according to annotation type
• Rank contributors • Reputation is main award • Allow users to donate money for good annotations
• Donated money will be awarded to the top annotators • The more and better annotations provided… • …the higher the reputation • …the higher the financial incentive
www.insemtives.eu 33
![Page 34: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/34.jpg)
Conclusion • Devising motivation methods for annotating Web
services is challenging • Different possibilities were/are being explored through
challenges – Mechanical Turk – Mashups Challenge – Long-Term Competition
• Users were closely kept in the development loop through OPD – Ensures that implemented features are usable – Keeps users engaged in a “community”-like way
www.insemtives.eu 34
![Page 35: INSEMTIVES Tutorial ISWC2011 - Session3](https://reader034.vdocuments.site/reader034/viewer/2022050804/547c5c1cb4af9fda158b50f7/html5/thumbnails/35.jpg)
Questions & Annotations
Thank You!
www.insemtives.eu 35