rigorous online courses - amazon web services...sessment vastly improves learning outcomes. by...
TRANSCRIPT
RigoRous online CouRses:Ontario Reach Ahead
A Crowdmark white paper / December 11, 2013
1
This white paper by Crowdmark:
• identifiestheproblemobstructingmassiveonlinecourses
fromachievingtheirpotential;
• advancesthesolutionthroughanewonlinecollaborative
gradingplatform;
• describesanonlinecoursescenarioincludingrigorous
assessmentwithaneffectivebusinessmodel;
• outlineshowgovernmentscanbooststudentachievement
whilesavingmillionsofdollars.
Thescenariodescribedhereisjustoneofmanywaysthatonline
collaborativegradingcanimprovetheglobaleducationsystem.
RigoRous online CouRses:Ontario Reach Ahead
2
ReachAheadopportunitiesinOntariopublic
secondaryschoolshavemanybenefits.Stu-
dentswhoearnauniversitycreditduringhigh
schoolaremore likely to finish university, and
can complete their degree sooner, reducing the
demand on the higher education system and
reducing student indebtedness.However,only
35%ofOntariosecondaryschoolsofferReach
Aheadopportunities,denyingstudentsinthe
other65%ofschoolsthebenefitsofearning
universitycredit.ThisviolatestheProvince’s
EquityandInclusiveEducationStrategy. Inthis
whitepaper,Crowdmarkpresentsanonline
coursescenariowithrigorousassessmentthat
enablestheMinistryofEducationtoprovide
ReachAheadopportunitiestoall secondary
schoolsinOntario.
Weuseasanexampleafirst-yearcalculus
class.CrowdmarkwillworkwithanOntariouni-
versityandsecondaryschoolstakeholdersto
createarichonlinecalculuscoursethatcovers
thematerialfromboththeOntarioGrade12
MCV4UCalculusexpectationsandafirst-year
universitycalculuscourse.Thecoursewillbe
broadcasttoeveryOntariosecondaryschool
twiceaweek,whileontheremainingthree
summary: BenefitstotheProvinceofOntario
Only 35% of Ontario secondary schools offer Reach Ahead
opportunities. In this white paper, Crowdmark presents an online course scenario with rigorous assessment that enables the
Ministry of Education to provide Reach Ahead opportunities to all
secondary schools in Ontario.
days,teachersintheparticipatingsecondary
schoolsworkwiththeirstudentsonhomework
andprovideadditionalacademicsupport. Stu-
dentsareassessedusingexamsdistributed
electronicallytoeachschool,printedand
writtenlocally,thenuploadedtoCrowdmarkfor
anonymization,randomization,anddistribution
totheparticipatingteachers,whogradethepa-
personlineusingCrowdmark,inhalftheusual
time.Attheendoftheyear,thetopstudents
(basedontheirassessments)areinvitedtoap-
plyforuniversitycreditatasignificantlyreduced
fee.Theuniversityhasaccesstothestudents’
portfoliosstoredinCrowdmarkandisableto
rapidlydetermineeligibilityforuniversitycredit.
3
Crowdmarkhasdevelopedaweb-basedplat-
formtosupporttherigorousassessmentofstu-
dentsrequiredforthesuccessofthisscenario.
Itcontainstheseessentialelements:
• Distribution Infrastructure: systems to
distributeexamstoandcollectexamsfrom
theparticipatingsecondaryschools;
• Team Dashboard:toolsforassemblingand
managingthemarkingbyteachersfrom
theparticipatingsecondaryschools;
• Crowdmarking Workflow:astreamlined
webprocessformarkingteamstocollabo-
rativelyevaluatestudentwork;
• Data Archive:anorganizedrepositoryfor
storingstudentworkandtheassociated
assessmentsofthatwork.
Thisisjustoneexampleofonlinecollaborative
grading.Onceitsvaluehasbeendemonstrat-
ed,onlineReachAheadopportunitiescouldbe
developedandmadeavailableinthesciences
andmanyotheracademicdisciplines,allowing
Ontariosecondaryschoolstudentstoenteruni-
versitywithuptoafullyearofvalidateduniver-
sitycredit,andgraduatewithafour-yeardegree
inthreeyears.Thisscenarioillustrateshow
theGovernmentofOntariocanbooststudent
achievementwhilereducingtimetodegreeand
savingtheProvincemillionsofdollars.
4
ReachAheadopportunitiesinOntariopub-
licsecondaryschoolshavemanybenefitsto
students.Studentswhoearnauniversitycredit
duringhighschoolhaveexpectationsandstudy
habitsthataremorealignedwithuniversity
requirements.Theyaremorelikelytofinishuni-
versityandcancompletetheirdegreesooner,
reducingthedemandonthehighereducation
systemandreducingstudentindebtedness.
Inthiswhitepaper,wedemonstratethatReach
Aheadopportunitiesprovide significant benefits
to the universityaswell.
Crowdmark presents a scenario with an effec-
tivebusinessmodelthatcreates a new revenue
streamforuniversities,andhasthepotential
to reduce the time to degree.ImagineaReach
Aheadopportunityofferedtoallsecondary
schoolsinOntario:forexample,afirst-yearcal-
culusclass.Crowdmarkwillworkwithuniversity
andsecondaryschoolstakeholderstocreatea
richonlinecalculuscoursethatcoversthema-
terialfromboththeOntarioGrade12MCV4U
Calculusexpectationsaswellasafirst-year
universitycalculuscourse.Thecoursewillbe
broadcasttoOntariosecondaryschoolstwice
aweek,whileontheremainingthreedays,
summary: BenefitstotheUniversity
Students who earn a university credit during high school
have expectations and study habits that are more aligned with university requirements.
Crowdmark presents a scenario ... that creates a new revenue
stream for universities, and has the potential to reduce the
time to degree.
teachersworkwithstudentsinclassonhome-
workandofferadditionalacademicsupport.
Studentsareassessedusingexamsthatmeet
universitystandardsforachievement.The
examsaredistributedelectronicallytoeach
school,printedandwrittenlocally,thenupload-
ed to Crowdmark for anonymization, random-
ization,anddistributiontoparticipatingteach-
ers,whogradethepapersonline.Attheendof
theyear,thetopstudents(basedontheiras-
sessments)willbeinvitedtoapplyforuniversity
credit,forafee.Theuniversitywillhaveaccess
toanarchivedcollectionofstudentworktouse
asthebasisforawardingcredit.
5
Crowdmarkhasdevelopedaweb-basedplat-
formtosupporttheefficient,rigorousassess-
mentofsecondaryschoolstudentsrequiredfor
thesuccessofthisscenario.Itiscomprisedof
theseessentialelements:
• Distribution Infrastructure: systems to
distributeexamstoandcollectexamsfrom
theparticipatingsecondaryschools;
• Team Dashboard:toolsforassemblingand
managingtheteamsofbothteachersfrom
theparticipatingsecondaryschoolsfor
markingexams,anduniversityassessors
foradjudicatingcreditawards;
• Crowdmarking Workflow:astreamlined
webprocessformarkingteamstocollabo-
rativelyevaluatestudentwork;
• Data Archive:anorganizedrepositoryfor
storingstudentworkandtheassociated
assessmentsofthatwork.
Thisisjustoneexampleofthedeploymentof
onlinecollaborativegrading.Oncefeasibility
has been demonstrated, Reach Ahead oppor-
tunitiescouldbemadeavailableinthesciences
andmanyotheracademicdisciplines,providing
asignificantrevenuestreamtotheuniversity.
6
Theemergenceofmassiveonlinecourse
platformshasexpandedtheabilityofeduca-
torstobroadcastrichmultimediainstruction
tohugeclassesofstudentsallovertheworld.
Participantsinthesecoursesdemonstrate
theirlearningbyansweringmultiplechoice
questionsinsidevideos,fillinginwebforms,
andparticipatingingiantonlineforumswith
otherstudents.
Comparedtowhathappensinasmallseminar,
studentsincurrentmassiveonlinecoursesdo
notreceiverichhumanresponsestotheirwork.
Thefeedbackchannelfromstudenttoinstructor
hasmuchlowerbandwidththanthebroadcast
channelfrominstructortostudent.Thecommu-
nicationsmismatchprobleminmassiveonline
coursesobstructstherigorouscertificationof
learningachievementandaccountsforthefail-
ureofMOOCbusinesseffortstodate.
TheInstructionalCommunications MismatchProblem
Collaborative online grading, ‘crowdmarking’, amplifies human
capacity to respond to and evaluate student work to
arbitrary scale. Crowdmark solves the instructional
bandwidth mismatch problem.
7
scenario: Ontario Reach Ahead
TheProvinceofOntariohas913public
secondaryschools.ByofferingInternational
Baccalaureate and AdvancedPlacement
programs,317oftheseschoolsofferaReach
Aheadopportunityallowinggrade12students
toearnfirst-yearuniversitycreditsthrough
studiesattheirhighschool.Studentsatthe
remaining596schoolsaredisadvantagedand
don’thaveReachAheadopportunities.This
situationisunfairandviolatesOntario’sEquity
andInclusiveEducationStrategy.
Theinequitygrowsmoresignificantwhenyou
considerthatstudentswhoearnauniversity
creditduringhighschoolaremorelikelytofinish
university.Thisresulthasbeenestablished
throughresearchandmakeslogicalsense:stu-
dentswhoperformuniversity-levelworkwhilein
highschoolhavetheirexpectationsandstudy
habitsalignedwithuniversitybeforetheyarrive
there.InOntario,thesituationwasexacerbated
bytheeliminationofGrade13OntarioAcadem-
ic Creditwhichproducedagapbetweenthe
K-12andpostsecondarycurriculathatcontin-
uestowiden.
Ontario’sinvestmentinhighereducationis
morelikelytoachievedesiredoutcomeswith
anexpansionofopportunitiesforstudentsto
earnuniversitycreditswhileinhighschool.
Students at 596 schools are disadvantaged and don’t have Reach Ahead opportunities.
317
596schoolswithReach Ahead
schoolswithoutReach Ahead
Reach Ahead opportunities in ontario secondary schools
8
Forsimplicity,let’sassumethat:
• Ontariohasexactly1,000publicsecond-
aryschools(insteadof931);
• Eachpublicsecondaryschoolhasamulti-
functionprinter/scannerandbasicinternet
connectivity;
• Eachpublicsecondaryschoolhasexactly
onegrade12calculusclass;
• Eachgrade12calculusclassistaughtby
exactlyoneteacher;
• Eachgrade12calculusclasshasexactly
25students;
• Eachclasswritestwomidtermsandone
finalexampersemester;
• Eachexamis10pageslongsoeachstu-
dentproduces60pagesofexamwork.
Proposal: ACalculusCourseto1,000Classrooms
We propose a deployment of an online calculus course that equalizes the Reach Ahead opportunity across the entire Province of ontario.
ThroughpartnershipwithanOntarioschool
board(forexample,theTorontoDistrictSchool
Board)oranOntarioindependentschool(for
example,theUniversityofTorontoSchools)
andanOntariouniversity(forexample,theUni-
versityofToronto),webuildateam-taughton-
linecourseandbroadcastrichmultimediacal-
culusinstructionfromtheuniversityinto1,000
Ontarioclassroomsusingtheweb.Thecourse
content,enrichedwithvideos,simulationsand
synchronousquizzesanddifferentiatedin-
struction,canbedesignedtocovertheOntario
Grade12MCV4UCalculusexpectationsaswell
asafirst-yearuniversitycourselikeMAT135 at
UofT.ThecourseisstreamedliveonTuesdays
andThursdays.OnMondays,Wednesdaysand
Fridays,the1,000teachersfliptheclassroom,
supplementingthecoursematerialwithgroup
problemsolvingsessionsandindividualized
instruction.
9
Proposal: ACalculusCourseto1,000Classrooms
exchange Between ontario High schools and the Host university
HostUniversity
HighSchool
10
ArchivedRigorousAssessmentof25,000StudentsArigorousfirst-yearuniversitycalculuscourse
willoftenhavetwomidtermexamsandafinal
exampersemester.Theseexamsinclude
questionsaskingstudentstoexpresstheirideas
throughwritingandcalculation.Eachofthese
examsisabout10pageslong,sothroughout
theacademicyeareachstudentcreates
approximately60exampagestobeassessed.
Theevaluationofresponsesfromstudents
requiresworkbyskilledgraders.Whengraders
giveformativefeedbacktostudents,themid-
termexamscreateopportunitiesforstudentsto
solidifytheirlearning.Thepioneeringresearch
byBlackandWilliamandastreamofsubse-
quentstudieshaveshownthatformativeas-
sessmentvastlyimproveslearningoutcomes.
Bytakingadvantageofcutandpaste,embed-
dinglinkspointingtolearningresourcesonthe
web,andallowingfortheexpressionofelegant
mathematicsincomments,thecollaborativeon-
linegradingplatformempowersteamsofmark-
erstogiverichformativefeedbacktostudents.
Itisadauntingtasktoimaginereading,evalu-
ating,scoringandannotating1,500,000exam
pages(60pages×25,000students).Thisisthe
equivalentofarowofbanker’sboxesaslongas
afootballfieldwitheachboxcontaining5,000
pages.Butwecanaccomplishthistaskby
leveragingtheteamof1,000remoteteachers
involvedusingonlinecollaborativegrading.
When graders give formative feedback to students, the
midterm exams create opportunities for students to
solidify their learning.
11
AssessmentDistributionInfrastructure
UsingCrowdmark,wecanrunandevaluatean
examinthereachaheadcalculusclass.With
LaTeX,MicrosoftWord,AdobeCreativeSuite
orwhatevertoolstheywishtouse,thelead
instructorsinthecourseoverseethecreationof
aPDFfilecontainingtheexamquestionsand
spacesfortheanswers.UsingtheCrowdmark
distributionsystem,25copiesofthisblank
ten-pageexamcanbesentelectronicallyin
theformofa250pagePDFfiletoeachofthe
1,000teachers.Eachpagedistributedthisway
byCrowdmarkhasauniquelyaddressable
graphicalcode.Theteachersprintandstaple
theexamsintobookletslocallyattheirschool.
Theexamsarethenwrittenbystudentsand
collectedbytheteachers.Staffattheschools
(orwithhelpfromoutsourcepartners)scan
the250pagesintodigitalimagesusingthe
school’sphotocopier/scanner.Onstandard
photocopiers,ascanjobof250pageswilltake
approximately20minutes.Thedigitalimages
fromeachschoolareuploadedintoCrowdmark
androutedintoadatabaseformarkingusing
thegraphicalcodesappearingoneachpage.
Rigorous online Course Assessment Distribution
Universityprofessorsendsdigitalassignmentstohighschools
Highschoolteachersprintassignmentsanddistributepaperstostudents
Studentscompleteassignmentsandhand them in to teachers
P
s s s s s s sss
T T T
s
T
P
HighschoolteachersscananduploadassignmentstoCrowdmark.
T
12
Crowdmarking
Eachofthe1,000teachersareinvitedtocollab-
orativelygradetheexam.Thistaskisorganized
usingtheteammanagementdashboard.Each
teachergradesthesamenumberofexamsas
s/hehasstudents,sothereisnoincreasein
workload.Butinsteadofmarkingtheexams
writtenbytheirownstudents,teachersgrade
arandomsampleofexamsfromacrossOn-
tario.Comparedtothetraditionalred-pen-on-
papermethod,thecrowdmarkingworkflowis
streamlined.Inexperimentstodate,Crowdmark
hascutgradingtimeby50%ormore.Graders
can“write”onthestudentexamimagesusinga
mouseorstylus.Graderscanleaverichforma-
tiveassessmentfeedbackcommentswithweb
linksandmathematicalexpressionsparsedby
TeXandMathjax.
Afterthegradingoftheexamisfinished,
theleadinstructorselectronicallyreturnthe
scannedimageswiththeassociatedscores,
commentsandannotationstoeachstudent.
Datafortheirownstudentsisalsomadeavail-
abletoindividualteachers.
Thisstreamlinedprocessallowstheteamto
reviewscoredisputes.Communicationschan-
nelsbetweenstudents,teachers,andparents
(ifappropriate)promoteinformationexchanges
anchoredontheassessmentandbuildnew
opportunitiesforlearningandengagement.
13
ArchivedPortfolio
Theexamimagesandassociatedfeedback
fromthemarkingteamarestoredinthecloud.
Theformativeassessmentfromallpreviousex-
amsremaininstantlyaccessibletostudentsas
theystudy.Theteachershaveaccesstoeach
student’sportfolioofwork.Thefeedbackgiven
bythemarkingteamsisvisibletothecentralin-
structionalteam.Byusingthecommunications
channelsandreferencingtheportfolioofstu-
dentworkandassessment,exchangesamong
thecentralinstructionalteamandamongthe
1,000remoteteacherscanimprovethequality
ofinstructionandpromotebetterGrade12to
universitycurriculumalignment.
Theportfoliobelongingtoeachstudentin-
cludesscannedimagesofeverypageoftheir
workinthecourseandthescores,comments,
andannotationsgivenbygradersoneach
page.Thisrecordenablestheuniversityto
makeadetailed,evidence-basedevaluationof
studentperformanceaspartoftheprocessfor
decidingwhethertoissuecredit.Theuniversity
couldchoosetohidethescoresof1,000teach-
ersandcarryoutacompleteinternalevaluation
ofthestudent’swork.Attheotherextreme,
theuniversitycouldchoosetocompletelytrust
theevaluationskillsofthe1,000teachers.By
buildingsystemstoevaluateandtraincrowd-
markers,Crowdmarkempowerstheuniversity
totailortheirreviewofthestudentportfoliopro-
ducingmorerigorousdecisionsaboutissuing
credit.Theportfolioofstudentworkalsocon-
tributestoamoreequitable,evidence-based
decisionprocessfordeterminingwhethercredit
canbetransferedtoanotheruniversity.
Using the communications channels and referencing the portfolio of student work and assessment ... can improve
the quality of instruction and promote better Grade 12 to
university curriculum alignment.
14
BusinessModelforReachAheadCourses
As a point of reference, and based on FFTE and
OntarioBasicIncomeUnits,traditionalyear-long
creditsattheUniversityofTorontocostabout
$2,800.Wedescribeabusinessmodelinwhich
Grade12studentsinthemassiveonlineReach
Aheadcalculuscourseearncreditsfor$100,
amassivedecreaseof96.5%ofthetraditional
cost.Studentswhoperformwell,forexamplein
thetop20%oftheclass,asmeasuredbytheir
scoresontheexamsaregiventheoptionto
submittheportfoliooftheirworktotheuniversi-
tyforconsiderationforcredit.Thecosttomake
therequestis$100andthe$100iskeptbythe
universityifitdeterminesthatacreditshould
beissued.Ifthecreditisnotissued,the$100
paymentcanbereturnedtothestudent.Ifthe
top-performing20%oftheclassexercisesthis
option,theuniversityreceives$100from5,000
students,generatingrevenueof$500,000from
onecalculuscourse.
Wecanestimatethecostsofrunningthe
course(seethefollowingpage).Notethatthe
salaryofthe1,000teachers,thesalaryofthe
centralinstructionalteam,thepastinvestment
inhardwareandsoftwareinfrastructure,the
pastinvestmentinphysicalinfrastructurelike
buildingsandclassroomsareleveragedinthis
scenarioanddonotrepresentnewcosts.The
mainadjustmentistochangefromanessential-
lyisolatedsetof1,000highschoolclassrooms
generallyfollowingacurriculumdefinedbythe
Provincetoanewcollaborativeworkflowof
1,000classroomsworkingtogetherinpartner-
shipwithauniversityandtheProvince.
Students earn credits for $100, a massive decrease of 96.5% of the
traditional cost.
15
Cost Estimates• Theonlinecoursecontentneedstobedesignedandprepared.
• Eachstudentwrites6examsperyear,producing60pagesofworktobeevaluated.
• 25,000students×60pages=1,500,000totalpages
• Thosepagesneedtobeprintedandscanned(wehaveaquantifiedsolution).
• Thepageswrittenbystudentsinthetop20%needsubsequentreviewbyuniversitystaff.
Quantifications
1. Coursepreparationcost:$100,000(one-time-only)*
2. Papercost:$12,900†
• Acaseofpapercontaining10reamshas5,000pages.
• Eachboxcost$43atOfficedepot.300boxesarerequired.
3. Paperprintingcost:1,500,000×$0.03=$45,000†
4. Outsourcescanning:1,500,000×$0.08=$120,000†
5. Crowdmarkcosts:$5/student×25,000students=$125,000
6. ReviewofTop20%:5,000×60=300,000pages=$80,000‡
• ExperimentsatUofTindicatecrowdmarkerscanreviewcalculusexampagesat150pages
perperson-hour.
• 300,000pagesreviewedat150pagesperhourwilltake2,000hours.
• UofTgraduatestudentTAscost$40/hour.
Total Cost, Year 1 (estimate):$482,900(Subsequentyearswillcostsignificantlyless.)
Representstheonetimeonlycostsofproducinganonlinecourseincludingvideoproductionandtechnicalsupport.Afterthecontentassetsareproduced,laterversionsofthecoursewillbelessexpensivetoproduce.
Canpotentiallybeviewedaspartofthenormalworkflowofthe1,000schoolsandthereforemaynotbeactualcosts.
Estimatedundertheassumptionthateverypagewrittenbystudentsinthetop20%receivesadetailedreviewbyUofTgraduatestudentteachingassistants.Thislevelofreviewmaynotbenecessaryandwillbecomeunnecessaryasthetrainingandevaluationsystemsofthecrowdmarkersdevelopinsophistication.
*
†
‡
16
CascadingBenefitsTherigorousonlinecoursestrategyoutlined
aboveisnotrestrictedtomathematicscourses.
Thesameapproach—leveragingonlinecollab-
orativegradingtoscalerigorousassessment—
canbeusedinothersubjects.Imagineifthis
strategyweresuccessfullyimplementedacross
allmajordisciplines.Wecanenvisionafuturein
whichamajorityofperseveringfirst-yearstu-
dentsarriveatuniversitywithayearofcourse-
workalreadycompleted.Thisshortensthetime
todegreeandopensupcapacityinthehigher
educationsystem.
We can envision a future in which a majority of persevering
first-year students arrive at university with a year of
coursework already completed.
Universitiescanbegintodevotemoreoftheir
limitedresourcestowardadvancedcoursesand
researchefforts.Theportfolioofstudentwork
andalltheassociatedfeedbackfrommarkers
allowsentitieslikeONCAT and ECTStofacilitate
credittransferdecision-makingbasedonmore
thoroughevidenceofstudentachievement.
ConclusionWehavedemonstratedthatOntariocancost-
effectivelyprovidereachaheadcreditsto
studentsProvince-wide.Theriskisminimal,as
theapproachleveragesexistinginvestmentsin
educationalinfrastructureandusesaproven
methodforscalingtheassessmentofstudent
learning.ItprovidestheopportunityforOntario
toimprovestudentachievementandstudent
successacrossourentireeducationalsystem.
17
AboutCrowdmarkCrowdmarkisaneducationtechnologycom-
panybasedinToronto.Crowdmark’sultra-scal-
ableassessmentplatformandbadgedlabor
marketenablesevaluatingandcertifyinglearn-
ingonamassivescale.Theglobaleducation
explosionlacksscalablemethodstocertify
achievement.Evaluatingstudentworkcurrently
involvespagingthroughpilesofpaperorusing
ineffectivemultiple-choiceexams.Crowdmark
streamlinesdocumentassessmentandmakes
iteasyforinstructorstorapidlymarktheworkof
theirownstudents,ortoassembleandmanage
teamsofqualifiedgradersforlargeclasseswith
hundredsorthousandsofstudents.
Contact Crowdmark:
JamesColliander
+1(416)8874833