making better use of the crowdbut microtasks are easy to share with collaborators without the need...
TRANSCRIPT
MakingBetterUseoftheCrowd
JennWortmanVaughanMicrosoftResearch
Afewdisclaimers…
Aretherebetterwaystomakeuseofthecrowd?
Whatotherproblemscanthecrowdsolve?
1. DirectApplicationstoNLPandMachineLearning
2. HybridIntelligenceSystems
3. LargeScaleStudiesofHumanBehavior
Part1:ThePotentialofCrowdsourcing
“Crowd”guitar
man
Part2:TheCrowdisMadeofPeople
• Whatmotivatesworkers?• Areworkersindependent?• Areworkershonest?
Whatdoesthisteachusabouthowtoeffectivelyinteractwithcrowd?
Hint:Berespectful.Beresponsive.Beclear.
Extensivenotes,slides,andeventuallyvideoat
http://www.jennwv.com/projects/crowdtutorial.html
Part1:ThePotentialofCrowdsourcing
1. DirectApplicationstoNLPandMachineLearning
2. HybridIntelligenceSystems
3. LargeScaleStudiesofHumanBehavior
ThePotentialofCrowdsourcing
GeneratingLabeledData
Learner
Learner“dog”
“cat”
“dog” “cat”
“cat” “cat”
Aggregationofnoisylabels
Learner“dog”
“cat”
TrainedModel
Aggregationofnoisylabels
“dog” “cat”
“cat” “cat”
“cat”
Learner“dog”
“cat”
TrainedModel
Aggregationofnoisylabels
“dog” “cat”
“cat” “cat”
Usedtoannotatemedicalimages,labeltext,extractandlabelfeatures ofscenes.
Inspiredhugeamountsofalgorithmicworkonaggregation.
AggregatingLabelswithEM• Input:Worker-generatedlabelsforeachinstance
• Calculateaninitialestimateofeachinstance’slabelbasedonasimplemajorityvote
• Repeatuntilconvergence:– Treatingthecurrentlabelestimatesastruth,estimateeachworker’squality
– Treatingthequalityestimatesastruth,calculatethemostlikelylabelforeachinstance
• Output:Oneaggregatedlabelforeachinstance[Dawid andSkene,1979]
AggregatingLabelswithEM
• Noguaranteesonoptimality,buttendstoworkprettywellinpractice
• Manyrecentvariantshavebeenproposedtoincorporatethevaryingdifficultylevelsofinstances,workerexpertise,theexistenceof“gold”tasks,etc.
[Dawid andSkene,1979]
BeyondSimpleLabels:CrowdTranslation
• Crowdworkers areaskedto– Translatesentencesfromonelanguagetoanother– Editotherworkers’translationstomakethemmorefluentandgrammatical
– Rankthequalityoftheresultingtranslations
• Canusemachinelearningtopredictthehighestqualitytranslationbasedonsentence-levelfeatures,worker-levelfeatures,andranks
[Zaidan andCallison-Burch,2011]
GeneratingSimilarityMeasures
[Gomesetal.,2011]
GeneratingSimilarityMeasures
flags noflags[Gomesetal.,2011]
GeneratingSimilarityMeasures
Democrats Republicans[Gomesetal.,2011]
CrowdClustering
[Gomesetal.,2011]
Bayesianmodel
CrowdsourcingforEvaluation
cheesekalebreadsteak
mushroompizza...
electionsenatebill
delegatepresidentproposal
...
EvaluatingTopicModels
Tobeusefulfordataexplorationorsummarization,topicsmustbehuman-interpretable!
[Changetal.,2009]
EvaluatingTopicModels
mushroom,kale,cheese,bread,election,steak
workeraccuracy
human-interpretability
Previousmeasuresofsuccess(e.g.,loglikelihoodofheld-outdata)donotimplyinterpretability!
[Changetal.,2009]
Word intrusiontask:
HumanDebugging
HumanDebugging
• Semanticsegmentation:partitionanimageintosemanticallymeaningfulparts,labeleachpart
[Parikh&Zitnick,2011;Mottaghi etal.,2013]
“cat”
HumanDebugging
• Semanticsegmentation:partitionanimageintosemanticallymeaningfulparts,labeleachpart
Whichcomponentistheweakestlink?
segmentclassifier
supersegmentclassifier
sceneclassifier
shapeprior
objectdetector
CRFmodel
[Parikh&Zitnick,2011;Mottaghi etal.,2013]
HumanDebugging
segmentclassifier
supersegmentclassifier
sceneclassifier
shapeprior
objectdetector
CRFmodel
[Parikh&Zitnick,2011;Mottaghi etal.,2013]
HumanDebugging
segmentclassifier
supersegmentclassifier
sceneclassifier
shapeprior
objectdetector
CRFmodel
[Parikh&Zitnick,2011;Mottaghi etal.,2013]
HumanDebugging
segmentclassifier
supersegmentclassifier
sceneclassifier
shapeprior
objectdetector
CRFmodel
[Parikh&Zitnick,2011;Mottaghi etal.,2013]
HumanDebugging
segmentclassifier
supersegmentclassifier
sceneclassifier
shapeprior
objectdetector
CRFmodel
[Parikh&Zitnick,2011;Mottaghi etal.,2013]
Humanslessaccurateattask,butsystemperformancestillimproved
1. DirectApplicationstoNLPandMachineLearning
2. HybridIntelligenceSystems
3. LargeScaleStudiesofHumanBehavior
ThePotentialofCrowdsourcing
HybridIntelligenceforSpeechRecognition
Crowd-BasedClosedCaptioning
Isitpossibletoprovidereal-timeclosedcaptioningoflectures,meetings,orotherday-to-dayconversations?
[Lasecki etal.,2012]
Thesystemmergesreal-timepartialinputs fromdynamic,untrainedcrowdstooutperformindividuals
Crowd-BasedClosedCaptioning
[Lasecki etal.,2012]
HybridIntelligenceforScheduling
Cobi:Communitysourced Scheduling
[projectcobi.com]
Abigconstrainedoptimizationproblemwithnoaccesstotheconstraints!
1. Committeesourcing 2. Authorsourcing
3. Scheduling 4. Attendeesourcing
[projectcobi.com]
Authorsourcing
crowdsourced clustering!
[projectcobi.com]
87%responserate!
Scheduling
[projectcobi.com]
Thesystemsolvesanoptimizationproblemtoproposeaschedule,butchairsretaincontrol.
HybridIntelligenceforWriting
TheSelfsourcing Process
1. Collectcontent
2. Organizecontent
3. Turncontentintowriting
[Teevan etal.,2016]
CollectContent
The MicroWriter breaks writing into microtasks.
Collaborative writing typically requires coordination.
Microtasks can be done while mobile.
Structure turns big tasks into small microtasks.
Microtasks can be shared with collaborators.
Collaborators can be known or crowd workers.
People have spare time when mobile.
Microtasks make it easy to get started.
[Teevan etal.,2016]
OrganizeContent
collaboration
microtask
mobile
The MicroWriter breaks writing into microtasks.
Collaborative writing requires coordination.
Microtasks can be done while mobile.
Structure turns big tasks into small microtasks.
Microtasks can be shared with collaborators.
Collaborators can be known or crowd workers.
People have spare time when mobile.
Microtasks make it easy to get started.
[Teevan etal.,2016]
TurnContentintoWriting
Collaborative writing typically requires coordination, but microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues or paid crowd workers.
[Teevan etal.,2016]
Collaborative writing requires coordination.
Microtasks can be shared with collaborators.
Collaborators can be known or crowd workers.
collaboration
TurnContentintoWriting
Collaborative writing typically requires coordination, but microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues or paid crowd workers.
Structure makes it possible to turn big tasks into a series of smaller microtasks. For example, the MicroWriterbreaks writing into microtasks. These microtasks make the larger task easier to start.
People have spare time when mobile, and these micromoments are ideal for doing microtasks.
[Teevan etal.,2016]
TheSelfsourcing Process
1. Collectcontent
2. Organizecontent
3. Turncontentintowriting
• Steps2&3couldbedownbycrowdworkers,traditionalML/AIapproaches,oracombination
• Authortakesfinalpass,noneedforperfection
Crowdsourcing
[Teevan etal.,2016]
HybridIntelligenceinIndustry
1. DirectApplicationstoNLPandMachineLearning
2. HybridIntelligenceSystems
3. LargeScaleStudiesofHumanBehavior
ThePotentialofCrowdsourcing
UserStudiesforSecurityResearch
HowwelldoInternetusersunderstandsecurityrisks?
Whotriestoguesspasswords?
Only14%mentionedbothstrangersand familiarpeopleasthreats
p@ssw0rd pAsswOrdvs.
[Uretal.,2016]
UserStudiestoImprovetheCommunicationofNumbers
[Barrioetal.,2016]
Perspectives
• IsaonehundredbilliondollarcuttotheUSfederalbudgetbigorsmall?
• Onehundredbilliondollarsisabout...– 3%ofthe2015USfederalbudget– 1/6ofannualUSspendingonmilitary– 30%ofthenetworthofBeyoncé– $5foreverypersoninNewYorkstate
[Barrioetal.,2016]
SixmonthsofNewYorkTimesfrontpagearticles
Workersratedotherworkers’perspectivesforhelpfulness
Chosethehighest-ratedperspectives
64quoteswithmeasurements
370crowd-generatedperspectiveswithincentivesforquality
[Barrioetal.,2016]
Step1:PerspectiveGeneration
PerspectiveExamples
• TheOhioNationalGuardbrought33,000gallons ofdrinkingwatertotheregion.
• Toputthisintoperspective,33,000gallonsofwaterisaboutequaltotheamountofwaterittakestofill2averageswimmingpools.
[Barrioetal.,2016]
PerspectiveExamples
• Theyalsorecommendedsafetyprogramsforthenation’sgunowners;Americansownalmost300millionfirearms.
• Toputthisintoperspective,300millionfirearmsisabout1firearmforeverypersonintheUnitedStates.
[Barrioetal.,2016]
Step2:PerspectiveExperiments
• Randomizedexperimentsrunon3200+subjectsonAMTtotestthreeproxiesofcomprehension– Recall– Estimation– Errordetection
• Supportfoundforthebenefitsofperspectivesacrossallexperiments– Example:55%rememberednumberoffirearmsinUSwithperspective,only40%without
[Barrioetal.,2016]
UserStudiesforOnlineAdvertising
TheCostofAnnoyingAds
[Goldsteinetal.,2013]
Advertiserspaypublisherstodisplayads,butannoyingadscostpublisherspageviews.
Howmuchdoannoyingadscostpublishersindollars?
vs.
TheCostofAnnoyingAds
[Goldsteinetal.,2013]
Step1:Usethecrowdtoidentifyannoyingads.
vs.
GoodAds
[Goldsteinetal.,2013]
BadAds
[Goldsteinetal.,2013]
Step2:EstimatetheCost•Workersaskedtolabelemailasspamornot• Showngood,bad,ornoads;paidvaryingamountsperemail• Howmuchmoremustaworkerbepaidtodothesametaskswhenshownbadads?
[Goldsteinetal.,2013]
Step2:EstimatetheCost
• Goodadsleadtoaboutthesamenumberofviews(emailsclassified)asnoads
• Costsmorethan$1extratogenerate1000viewsofbadadsinsteadofnoadsorgoodads
• Takeaway:Publisherslosemoneybyshowingbadadsunlesstheyarepaidsignificantlymoretoshowthem
[Goldsteinetal.,2013]
1. DirectApplicationstoNLPandMachineLearning
2. HybridIntelligenceSystems
3. LargeScaleStudiesofHumanBehavior
SummaryofPart1
Part2:TheCrowdisMadeofPeople
Traditionalcomputersciencetoolsletusreasonaboutprogramsrunonmachines(runtime,scalability,correctness,...)
Whathappenswhentherearehumansintheloop?
Needamodelofhumanbehavior.(Aretheyaccurate?Honest?Dotheyrespondrationallytoincentives?)
Wrongassumptionsleadtosuboptimalsystems!
“ButIonlywanttousecrowdsourcing togeneratetrainingdataorevaluatemymodel.”
Understandingthecrowdcanteachyou– Howmuchtopayforyourtasksandwhatpaymentstructuretouse
– Howmuchyoureallyneedtoworryaboutspam– Howandwhytocommunicatewithworkers–Whetheryourlabels/evaluationsareindependent– Howtoavoidcommonpitfalls
TheCrowdisMadeofPeople
• Crowdworker demographics• Honestyofcrowdworkers• Monetaryincentives• Intrinsicmotivation• Thenetworkwithinthecrowd
Bestpractices!Tipsandtricks!
CrowdsourcingPlatforms
AmazonMechanicalTurk
Workers Requesters
AlternativePlatforms
• GermanplatformwithmanyEuropeanworkersofferingsupportfortranslationandwebresearchplusmobilecrowdsourcing
• OffersenterprisesolutionsforbusinesseswithAI/dataneeds(searchrelevanceevaluation,sentimentanalysis,dataclassification)
AlternativePlatforms
• Marketplaceforfreelancerswithlargerjobslikewritingarticlesordesigningwebsites
• UK-basedplatformfocusedonconnectingresearcherswithsubjectsforexperiments
Crowdworker Demographics
DemographicsofMechanicalTurk
[mturk-tracker.com]
DemographicsofMechanicalTurk
[mturk-tracker.com]
• 70-80%US,10-20%India
• Roughlyequalgendersplit
• Median(reported)householdincome:– $40K-$60KforUSworkers– Lessthan$15KforIndianworkers
• Canbebigchangesdependingontimeofday
Areworkersdishonest?
ExperimentalParadigm
• Askparticipantsaboutdemographics– Sex,Age,Location,Income,Education
• Askparticipantstoprivately rolladie(orsimulateitonanexternalwebsite)andreporttheoutcome
payment=$0.25+($0.25*roll)
• Ifworkershonest,meanreportedrollshouldbeabout3.5...Whatdoyouthinkthemeanwas?
[Suri etal.,2011]
Baseline
• Averagereportedrollhigherthanexpectation– M=3.91,p <0.0005
• Playersunder-reportedonesandtwosandover-reportedfives
• Butmanyworkerswerehonest!
• SimilartoFischbacher &Huesi labstudy
Roll
Proportion
0.00
0.05
0.10
0.15
0.20
0.25
1 2 3 4 5 6
[Suri etal.,2011]
Thirtyrolls
• Overall,muchlessdishonesty
• Averagereportedrollmuchclosertoexpectation– M=3.57,p <0.0005
• Only3of232reportedsignificantlyunlikelyoutcomes
• Only1wasfullyincomemaximizing(allsixes)
• Whyisthisthecase?
Roll
Proportion
0.00
0.05
0.10
0.15
1 2 3 4 5 6
[Suri etal.,2011]
DishonestyCanAddUp
• “Areyoutheparentorguardianofachildwithautism?”– 4.3%ofparticipantssaidyesincontrol– 7.8%ofparticipantssaidyeswhentoldthatthiswasaprescreeningtest forafurtherstudy
• Seemslikeasmalldifference,butwouldleadto(atleast)45%impostersinthesubsequentstudy!
[ChandlerandPaolacci,2011]
Takeaways&RelatedBestPractices
• Mostworkersarehonestmostofthetime.
• Butsomearenot.Youshouldstillusecaretoavoidattacks.
• Workersmaydeceiverequesterstogainaccesstowork.Prescreeningshouldbedonewithcare,ideallyaspartofaseparatetask.
MonetaryIncentives
Howmuchshouldyoupay?
Ausefultrick:• Pilotyourtaskonstudents,colleagues,orafewworkerstoseehowlongitgenerallytakes.
• UsethattomakesureyourpaymentsworkouttoatleasttheUSminimumwage.
Benefits:• It’sthedecentthingtodo!• Ithelpsmaintaingoodrelationshipswithworkers.
Canperformance-basedpaymentsimprovethequalityofcrowdwork?
Proofreadthistext,earn$0.50
Earnanextra$0.10foreverytypofound
[Hoetal.,2015]
PriorWorkonCrowdPayments
–Payingmoreincreasesthequantityofwork,butnotthequality[MW09,RK+11,BKG11,LRR14]–PBPsimprovequality[H11,YCS14]–PBPsdonotimprovequality[SHC11]–Bonussizesdon’tmatter[YCS13]
[Hoetal.,2015]
Performance-BasedPayments
Weexplorewhen,where,andwhy performace-basedpaymentsimprovethequalityofcrowdworkonAmazonMechanicalTurk.
[Hoetal.,2015]
CanPBPs work?
• Warm-uptoverifythatPBPs canleadtohigherqualitycrowdwork onsometask.
• TestwhetherthereexistsanimplicitPBPeffect:workershavesubjectivebeliefsonthequalityofworktheymustproducetoreceivethebasepayment,andsoalreadybehaveasifpaymentsare(implicitly)performance-based.
[Hoetal.,2015]
CanPBPs work?• Task:Proofreadanarticleandfindspellingerrors. • Werandomlyinsert20typos
• sufficiently->sufficently• existence->existance• …
• Usefulproperties:• Qualityismeasurable• Exertingmoreeffort->betterresults
[Hoetal.,2015]
CanPBPs work?Basepayment:$0.50;Bonuspayment:$1.00
ThreeBonusTreatments:• NoBonus: nobonusormentionofabonus• BonusforAll: getthebonusunconditionally• PBP: getthebonusifyoufind75%of
thetyposfoundbyothers
TwoBaseTreatments:– Guaranteed: guaranteedtogetpaid– Non-Guaranteed: nomentionofaguarantee
[Hoetal.,2015]
CanPBPs work?
• Resultsfrom1000uniqueworkers
• Guaranteedpaymentshurt(implicitPBP)
• PBPs improvequality
• Unlikeinpriorwork,payingmorealsoimprovesquality
[Hoetal.,2015]
UnderwhatconditionsdoPBPs work?Bonusthreshold(585uniqueworkers)• $0.50base+$1.00bonusforfindingXtypos
Ctrl 5T 25% 75% All
• PBPs workforawiderangeofthresholds
• Subjectivebeliefs(5typosvs.25%oftypos)canimprovequality
[Hoetal.,2015]
Bonusamounts(451uniqueworkers)• $0.50base+$X bonusforfinding75%of typos• PBPs workaslongasthebonusislargeenough
●
●
●
●
11
12
13
14
0.00 0.25 0.50 0.75 1.00
Bonus Amount
Typo
s Fo
und
couldexplainShawetal.,2011couldexplainYinetal.,2013
[Hoetal.,2015]
UnderwhatconditionsdoPBPs work?
WhichtasksdoPBPs workon?
• Whatpropertiesofataskleadtoqualityimprovementsfromperformance-basedpay?
• Somepilotexperimentsonaudiotranscriptionsuggestedthat– PBPs improvequalityforeffort-responsivetasks– Itisnotalwaysstraight-forwardtoguesswhichtasksareeffort-responsive
[Hoetal.,2015]
WhichtasksdoPBPs workon?
[Hoetal.,2015]
Takeaways&RelatedBestPractices
• AimtopayatleastUSminimumwage.Pilotyourtasktofindouthowlongittakes.
• Performance-basedpaymentscanimprovequalityforeffort-responsivetasks.Pilottochecktherelationshipbetweentimeandquality.
• Bonuspaymentsshouldbelargerelativetothebase.Thepreciseamountandprecisecriteriaforreceivingthebonusdon’tmattertoomuch.
IntrinsicMotivation
WorkThatMatters• Threetreatments:– control:nocontextgiven– meaningful: toldtheywerelabelingtumorcellstoassistmedicalresearchers
– shredded: nocontext,toldworkwouldbediscarded
• Meaningful->quantity up,butquality similar• Shredded->quality down,butquantity similar
[ChandlerandKapelner,2013]
Gamification
[vonAhn andDabbish,2004]
Gamification
[vonAhn,Kedia,andBlum,2006]
Takeaways&RelatedBestPractices
• Workersproducemoreworkwhentheyknowtheyareperformingameaningfultask,butthequalityoftheirworkmightnotimprove.
• Gamificationcanalsoincreaseproductivity.Wellcalibratedtimedresponsesandscorekeeping(withorwithouthighscorelists)canbothincreaseenjoyment.
TheCommunicationNetworkWithintheCrowd
Implicitassumption:Crowdworkersareindependent
[Yinetal.,2016]
Inrealityworkerstalkandcollaborate
Recreatesocialconnectionsand
supportM.L.Gray,S.Suri,S.S.AliandD.Kulkarni.TheCrowdisaCollaborativeNetwork.CSCW 2016
N.Gupta,D.Martin,B.V.Hanrahan andJ.O’Neil.Turk-lifeinIndia.Group 2014
Helpeachotherwithadministrativeoverhead
Ming’stasksaregreat!
Sharetasksandreputableemployers
Ethnographicfieldstudiesshowthat crowdworkers...
[Yinetal.,2016]
ACommunicationNetwork
Whatisthescale? Whatisthestructure? Howisitused?[Yinetal.,2016]
Ourgoal:Opentheblackboxofcrowdsourcing tomapthecommunicationnetworkofcrowdworkers
[Yinetal.,2016]
Whyisitchallenging?
ThenetworkisnotaccessiblefromtheAPIsowecan’tsimplydownload,crawl,orscrapeit!
Wanttomapthenetworkinawaythat#1Elicitsonly“true”edges#2Elicitsasmanytrueedgesaspossible#3Preservesworkers’privacy
[Yinetal.,2016]
AWebApp
• Workers self-report their connections
• Providessomevaluebacktotheworkerssothatit’sintheirbestinteresttoreportasmanytrueconnectionsaspossible
[Yinetal.,2016]
5268connections
10,354 workers(roughlyacensusofMechanical Turk[Stewartetal.2015])
[Yinetal.,2016]
1,389(13%)connectedworkers
Onaverage,workerscommunicatewith7.6 others
Maxdegreeis321
[Yinetal.,2016]
Largestcomponentincludes994(72%) workers
[Yinetal.,2016]
ANetworkEnabledByForums
• 59% ofallworkersand83% ofconnectedworkersreportedusingatleastoneforum.
• 90%ofalledgesarebetweenpairsofworkerswhocommunicateviaforums,and86% arebetweenpairswhocommunicateexclusivelythroughforums.
[Yinetal.,2016]
ForumsCreateSubcommunities
Reddit HWTF MTurkGrind TurkerNation
Facebook MTurkForum[Yinetal.,2016]
Subcommunities AreDifferent
TopologicalStructure: Howtightlyconnectediseachsubcommunity?
TemporalDynamics: Dorelationshipsendureovertime?
CommunicationContent: Iscommunicationsocialorstrictlybusiness?
[Yinetal.,2016]
MeasuresofSuccess
Property Connected UnconnectedBeactive>1year 55% 46%
Useforums 83% 56%Master 11% 7%
Approvalrate 98.6% 97.4%
[Yinetal.,2016]
Connectedworkerswerealsomorelikelythanunconnectedworkerstofindourtaskearly.
TakeawaysandRelatedBestPractices
• Forumusageiswidespread.Forumsarethevirtual“watercoolers”ofcrowdworkers.
• Engagewithworkersonforums.Introduceyourself.Introduceyourtasks.
• Activelymonitorforumdiscussionaboutyourtask.Whenappropriate,requestthatworkersdonotdiscussyourtask.Monitoranyway.
• Becarefulaboutassumingindependence!
AdditionalBestPractices
MaintainGoodRelationshipswithWorkers
• Setasidetimetoactivelymonitoryourrequesteremailaccountandrespondtoquestions.
• Approveworkquickly.
• Avoidrejectingworkexceptinthemostextremeofcircumstances.
• Strivetobeanethicalrequester.
http://guidelines.wearedynamo.org
TipstoMakeYourProjectRunSmoothly
• Pilot,pilot,pilot!Testyourtaskonyourcollaborators,othercolleagues,andeventuallysmallbatchesofworkers.
• Iterateasmanytimesasneeded.
Ifyourememberoneslidefromthistalk,rememberthis!
TipstoMakeYourProjectRunSmoothly
• Createclearinstructions.Includequizquestionsifneeded.Pilotthemandcollectfeedback.
• Createanattractiveandeasy-to-useinterface.Pilotthistoo!
• Askworkersforfeedback.Askthemtoreportbugs.Conductexitsurveyswhenappropriate.Workersgenerallywanttohelp!
Thanks...
ToChien-Ju Ho,AndrewMao,JoellePineau,SidSuri,HannaWallach,andespeciallyMingYinforextensivediscussionsandfeedback
ToDanGoldstein,Chien-Ju Ho,JakeHofman,Roozbeh Mottaghi,SidSuri,JaimeTeevan,MingYin,Haoqi Zhang,andalloftheircollaboratorsfortheuseofmaterialfromtheirslides
Andtoallthepeoplewhosentmepointerstocoolresearch...thistutorialwasacrowdsourced effort!
Extensivenotes,slides,andeventuallyvideoat
http://www.jennwv.com/projects/crowdtutorial.html
[email protected]://jennwv.com@jennwvaughan