ku · learning elements, peer review discussions and peer assessment in the bsc/msc level summer...

u n i ve r s i t y o f co pe n h ag e n

Technology enhanced peer learning and peer assessment

Henriksen, Christian Bugge; Bregnhøj, Henrik; Rosthøj, Susanne; Ceballos Garcia, AlejandroRafael; Kaas, Henrik; Harker-Schuch, Inez Estelle P; Andersen, Ingelise; Larsen, Søren;May, Michael

Published in:Læring og Medier (LOM)

DOI:10.7146/lom.v9i16.24415

Publication date:2016

Document versionPublisher's PDF, also known as Version of record

Document license:CC BY-NC-ND

Citation for published version (APA):Henriksen, C. B., Bregnhøj, H., Rosthøj, S., Ceballos Garcia, A. R., Kaas, H., Harker-Schuch, I. E. P., ... May, M.(2016). Technology enhanced peer learning and peer assessment. Læring og Medier (LOM), 9(16).https://doi.org/10.7146/lom.v9i16.24415

Download date: 02. mar.. 2021

https://doi.org/10.7146/lom.v9i16.24415

https://curis.ku.dk/portal/da/persons/christian-bugge-henriksen(bd67e28c-5466-439a-a716-946fb14e8816).html

https://curis.ku.dk/portal/da/persons/henrik-bregnhoej(e422e6d2-44c8-4af2-b09f-53353779240b).html

https://curis.ku.dk/portal/da/persons/susanne-rosthoej(56c290d5-658a-41a6-8540-26ae4ba1b42d).html

https://curis.ku.dk/portal/da/persons/henrik-kaas(8eb7ce7c-4a94-4472-bb80-cad9af66ff04).html

https://curis.ku.dk/portal/da/publications/technology-enhanced-peer-learning-and-peer-assessment(48837b2b-ccfb-4a4f-886c-ddbefaa57dc9).html

https://doi.org/10.7146/lom.v9i16.24415

Læring & Medier (LOM) – nr. 16 - 2016 ISSN: 1903-248X

http://www.lom.dk 1

Technologyenhancedpeerlearningandpeerassessment

ChristianBuggeHenriksenPhD,AssociateProfessorDepartmentofPlantandEnvironmentalScience,UniversityofCopenhagen

HenrikBregnhøjPhD,E-learningConsultantCenterforOnlineandBlendedLearning,UniversityofCopenhagen

SusanneRosthøjPhD,AssociateProfessorDepartmentofPublicHealth,UniversityofCopenhagen

AlejandroCeballosMSc,AssistantLecturerDepartmentofPlantandEnvironmentalSciences,UniversityofCopenhagen


http://www.lom.dk 2

HenrikKaasMSc,SpecialconsultantITLearningCenter,UniversityofCopenhagen

InezHarker-SchuchMSc,AssistantLecturerDepartmentofPlantandEnvironmentalSciences,UniversityofCopenhagen

IngeliseAndersenPhD,AssociateProfessorDepartmentofPublicHealth,UniversityofCopenhagen

SørenLarsenMSc,E-learningProjectManagerDigital,UniversityCollegeSjælland

MichaelMayPhD,Part-timeTeacherDepartmentofDesignandCommunication,UniversityofSouthernDenmark

TheauthorshaveworkedtogetherforseveralyearsinanonlineandblendedlearningprojectattheUniversityofCopenhagen.


http://www.lom.dk 3

AbstractThis paper explores the application of learning designs featuring formalised andstructured technology enhanced peer learning. These include student producedlearningelements,peerreviewdiscussionsandpeerassessment intheBSc/MSc levelsummer course Restoration of European Ecosystems and Freshwaters (REEF), theMaster thesis preparation seminars for the Master of Public Health (MPH) and theMOOC course Global Environmental Management (GEM). The application of studentproduced learningelementsandpeer reviewdiscussions is investigatedbyanalyzingquotes from course evaluations and performing focus group interviews. Theapplication of peer assessment is investigated by analyzing the agreement of peerassessment between students assessing the same assignment. Our analyses confirmpreviousresearchonthevalueofpeerlearningandpeerassessmentandwearguethatthere could also be a huge benefit from developing learning design patterns thatfacilitateinformalpeerlearningandreinforceknowledgesharingpractices.

IntroductionAligned with the constructivist theory that knowledge development is cultivatedthroughexperience,peer-to-peerlearningoccurswhen ‘peer-related’studentsengagein information and knowledge exchange for the mutual benefit of all participatingparties; teachingand learning fromoneanotherasbothpartofaplannedcurriculumandinabroader‘experientiallearning’sense.Thefoundationofpeer-to-peerlearningoriginates in theearlyenculturationprocesswhereby individuals learn socio-cultural‘behaviours’and ‘values’ fromthoseclosest to them- their family, friendsandknownfamiliars (Vygotsky 1930-1934/1978). This enculturation forms the springboard forstronger peer-to-peer information and knowledge exchange that takes place duringpuberty when adolescents transfer respect-authority regimes from their parents tothose ina similar agegroupor ‘peers’ (Piaget,1932).This transferof authority is animportantstepforpeer-to-peerlearningaspeersreplacesignificantfamiliars(family)withtheirpeerstoobtainnewinformation,ideasandknowledge(ortomodify,expand,or reinforce existing knowledge). As the parental-child relationship untethers, thesignificance of the community of peers increases in the adolescents estimation - somuchsothattheindividualwillgladlysacrificerespectandstandingwitheldersratherthan risk jeopardising relationships within their peer-community (Burnett &Blakemore,2009).Thispathway,then,formsthebasisforself-determinedsocialisationand the individual's first forays into adulthood; corresponding to the ‘second criticalstage of intellectual development’ that heralds the commencement of abstractreasoning, critical thinking skills and multidimensional problem-solving. As theadolescentmoves into adulthood, the frameof ‘peers’ broadens to include aspects ofsimilar interests (hobbies, careers), social collectives (child-rearing, ideologies) andotherfamiliarconcepts-wideningtheexposureoftheadolescenttoabroaderarrayofexperience, knowledge and information at the very peak of their intellectualdevelopment.Individualsderivemeaningandtrajectory,too,fromthosearoundthem-lifestyle choices and ideologies are transferred amongst peers; affecting enormouschange and upheaval. Adolescents and young adults can be profoundly inspired by


http://www.lom.dk 4

actionsandidealsofpowerfulpeerstothepointwheretheywillsacrificetheirlivestodemonstrate their association (e.g. radicalisation). Inspired by the work of VygotskyandPiaget, Lave andWenger developed their theory of social learning, including theconceptof“situated learning”wherepeer learning is takingplace ina“CommunityofPractice” that can be both formal and informal (Lave and Wenger, 1990, Wenger,1998). InauniversitysettingaCommunityofPracticeis formedwhenstudentsteachandlearnfromeachothereitherinconnectionwithacourseoraproject.ItfollowsthatpeerlearninginsuchaCommunityofPracticetakeseveraldifferentforms:

• Formalpeerlearning,includingstudentproducedlearningelements• Informalpeerlearning,includingandexcludinginstructorfacilitation• Peerassessment,includingpeer-reviews

Formal peer learning is when the instructor requires students to take turns inteachingeachother.TheteachingcanhappenonlineorIRL,anditcantaketheshapeofa lecture,apeerreviewdiscussionorastudentcanguide fellowstudents throughanassignment. Formal peer-to-peer teaching also includes student produced learningelements, which can be texts, videos, quizzes/tests, presentations, etc. developed byindividuals or groups of students aimed as learning resources for other students tolearn from. The benefits of student produced learning elements as exemplified byHakkarainenetal.(2007)arethatstudentsbothintheprocessofdesigning,producingand engaging in developing case based teaching materials experienced an enhancedpositive emotional involvement in the learning process promoting the active andcontextualaspectsof learning. It is importanttonotethataskingstudentstoproducelearning elements for each other,might require that students invest substantial timeand effort in gaining e.g. technical competences, video production skills, etc. Theinstructorneed to considerwhether these competences and skills are important andmatchesthelearningobjectivesofthecourse.

Informal peer learning covers a number of different learning activities:‘Turn to your neighbour’ - In a learning environment, students often turn to theirneighbour and ask for clarification or explanations when they have not understoodsomething. This canhappenduringa lectureorwhenstudentsareworkingon classwork.Turningtoyourneighbouris,perhaps,themosteffortlessformofinformalpeer-to-peerteachinganddoesnot,usually,worktoconstructhighly-complexknowledge.

‘Think-pair-share’-Thismethodworksinformalclassroomenvironmentsbutembedsinformal peer-to-peer interactionwithin the class. Students are divided into groups,givena topicorproblem, told theymust ‘think’ about theproblem for someminutes(which promotes higher-quality responses) then they are to pairwith a partner anddiscussthetopicorsolution.Theinstructorthenasksafewstudentstoshare-therebyincreasingtheresponsesfromthestudentsaswellasthelevelofresponseandactivitywithintheclassroom.

Vicarious learning - informal interaction amongst peers that serves to improveknowledgeorunderstanding. Thisoccurswhen fellowstudents (with similar status)


http://www.lom.dk 5

share knowledge - but depends on the premise that fellow students have knowledgeworth sharing. It is characterised by ‘active listening and reflective thinking’ (Nehls,1995) and seeking to understand by being fully absorbed in the activity (Roberts,2010). The sharing often takes place through ‘reflection on practice encounters andrelating these through stories to fellow students’ (this is also referred to ‘culture-making by Bruner, 1986). This is often coupled with personal and emotionalexperiences,aswell.

Study partnerships or groups - planned or organised studying together can also beconsidered ‘informal’ as such activities exist externally to the formal learningenvironment of their instruction. These allow students to share their understandingand strengthen their knowledge through concentrated study and focus on coursematerial - exchanging their perceptions and reinforcing the context of the coursematerial.

Story-telling/boasting - this is,arguably, theoldest formof teaching in theworldandmanifests in both formal and informal peer-to-peer group settings. In nursing andmedicalschoolenvironments,practiceofproceduresisexplainedtoothers(oftenwithanelementofdrama)and themethodof theprocedure is explained - enabling thoselisteningto‘gain’knowledgeorunderstandingthroughthestory.

Reaching out to another student ‘expert’ - This is donewith note-taking and occurswhenamore-able(alsomoreadvancedormoresenior)studentisapproachedfor(oroffersto)helpaless-proficientstudent.Thisisinformalasitexistsexternallyfromtheformallearningenvironment.

Game-playing - playing quizzes or games with peers often aids in knowledgedevelopment.Peoplelearn‘byparticipation’inthegame-and,often,throughaprocessoftrialanderror.

Peerassessmentisanotherformofpeerlearningthat“canbedescribedgenerallyasaprocesswherebystudentsevaluate,orareevaluatedby, theirpeers”(vanZundertetal,2010),i.e.‘peers’beingtheirfellowstudents.Peerassessmentcantakedifferentformsinwhich studentsprovidequalitative feedback for eachother’swork (includingpeerfeedbackandpeerreview)and/orgivesummativegradesforwrittenassignmentsandoral presentations. Dependent on the context and the purpose of using peerassessment,itcanbedoneasalearningactivitywithopendiscussionsandevaluationor anonymously in amore closed format similar to that of peer review on academicpapers.Itiscrucialthatthepurposeandclearexpectationsofusingpeerassessmentisexplicitlycommunicatedtostudentsaswellasfellowinstructors.Theprimarywayofdoingthisisbypreparingclearassessmentcriteriae.g.asanassessmentrubric(Dochyetal.,1999).AirasianandRussell(2008)defineassessmentrubricsas:“Arubricisasetof clear expectations or criteria used to help instructors and students focus onwhat isvaluedinasubject,topic,oractivity”.Anassessmentrubricisamatrixofexplicitcriteriathatcanbeusedtoevaluatestudentperformance.


http://www.lom.dk 6

ObjectivesTheobjectivesofthispaperaretoinvestigateandevaluatelearningdesignsfeaturingtheapplicationofstudentproducedlearningelementsinpeer-to-peerteaching,andtheuseof formalizedpeer feedbackandpeer assessment inonline andblended learningcoursesattheUniversityofCopenhagenbyapplyingamixedmethodsapproach.

MethodologyMethodologicalconsiderationsEvaluationofpeer learningcanbedone inmanyways,dependingonwhatshouldbeevaluated. If theeffectsofpeer learningon learningoutcomeshouldbeevaluatedthemostdirectwaytodothiswouldbetoeitherincludeorexcludepeerlearningfromacourse and determine the difference in final grades. However, this approach isproblematicinauniversitysettingsinceallstudentsshouldreceivethesameteaching.Alternatively, questionnaires or focus group interviews could be performed to askstudentsabouttheirownperceptionsofpeerlearning.Thiswillnotdirectlydeterminethelearningoutcomefrompeerlearning,butinsteadgiveanideaaboutwhetherpeerlearningisabletocreateaformalorinformalcommunityofpracticethatwillfacilitatethe achievement of learning outcomes. If peer learning includes elements of peerassessment, it will be important that the homogeneity of the peer assessment isdeterminedinordertoverifywhetherthepeerassessmentisvalid.Inthisstudyweareapplyingamixedmethodsapproachwithquestionnaires, focusgroup interviewsandstatisticaldataanalysisdependingonwhatismostsuitableforthethreecoursesweareinvestigating.(SeeAppendixAforadescriptionofthecourse).

EvaluatingexperienceswithlearningdesignsfeaturingstudentproducedlearningelementsIn the Restoration of European Ecosystems and Freshwaters (REEF) course theexperiences with student produced learning elements were evaluated in 2011 byextracting data from the students’ evaluation and subsequent reflections in theclassroom.Weusedanonlinequestionnaireinordertogetthefeedbackfromstudentsontheuseofstudentproducedvideosinthecourse.Thefollowingsixquestionswithafive-level Likert scale and and an open-answer question was included to investigatehowstudentsperceivedthecreationofthevideosasalearningprocess:

1. Towhatextenthaveyouexperiencedtheuseofvideoasamotivatingfactorforyourworkwiththecourse?

2. In this course, the use of video to document and report the fieldwork waspreferablecomparedtohandinginaregularwrittenreport.

3. Theuseofvideotocommunicatescientificworkonthe internet isapowerfultoolthatstudentsshouldlearnhowtomasterforuseintheirfuturecareer.

4. The process of making a video instead of a written report gave me a betterunderstandingofthestudiedsubject.

5. ThevideosproducedinthiscoursehavegreatpotentialinhelpingrisingpublicawarenessaboutrestorationofecosystemsinEurope.


http://www.lom.dk 7

6. Theproductionofavideohelpedmetoworkindependentlyandstimulatedmycreativity

7. What do you think about using Youtube and other online tools in universityteaching?

In the subsequent classroom reflections the discussion focused on the followingquestions: (i) howwas it to use another format than a traditional written report todocumentnaturalsciencesfieldwork?(ii)whatisthepotentialforusingthevideosasinstructor assisted peer to peer teaching material and (iii) what is the potential forusing thesevideosasmaterial toraisepublicawarenessonrestorationof freshwaterecosystems?

EvaluatingexperienceswithlearningdesignsfeaturingpeerreviewdiscussionsIntheMasterofPublicHealth(MPH)experienceswithpeerreviewdiscussionsduringthesisseminarsin2012wereevaluatedwiththefollowingquestionnaire:

1. Doyourecommendthatweofferseminarstothe2ndyearMPH-studentsnextyear?Yes/no

Pleasefeelfreetocommentonyouranswer.Ifyes:

1. Whichthemesshouldweworkwithintheseminars?2. Howfrequentshouldwehaveseminars?3. Shouldtheseminarsbeautumnandspringoronlyspring?4. Wouldyoupreferseminarsintheeveningorweek-endsinsteadoftwoweek

days?5. Anysuggestionsforimprovingtheseminars?

Based on the answers to the answers to the questionnaire focus-group interviewsweresubsequentlymadewithformerandcurrentDanishMPH-students.

EvaluatingexperienceswithlearningdesignsfeaturingpeerassessmentIn the Global Environmental Management course experiences with peer assessmentwas evaluated by performing a statistical analysis of the homogeneity of the peerassessment following themethodology developed by Fleiss et al (2003). In the GEMcourse peer assessment was performed by using a rubric with a set of criteria forassessment,eachwithamarkingscalewithadescriptionof theadequacyneeded foreach particular mark. For example one of the criteria for assessing the Technologyessaywas:Givefrom1-4pointsdependingonhowwellthetechnologyisdescribed.Thescaleswereasfollows:

● 1point:Thispartofthequestionisnotanswered● 2points:Thedescriptionisunclearoritspurpose/functionseemsnottobe

understoodbytheauthor● 3points:Itisdescribed,butI'mindoubtwhetheritspurposeorfunctionis


http://www.lom.dk 8

understoodbytheauthor● 4points:Itiswelldescribed,clearlyunderstoodbytheauthor

ThedistributionofthecriteriaincludedintheanalysisofpeerassessmentintheGEMcourseisshowninTable1

Table1.DistributionofcriteriaincludedintheanalysisofpeerassessmentintheGEMcourse.

Essay: Trend Management Technology

Numberofcriteria: 2 4 3

Markingpointspercriteria: 1-4 1-3 1-4

Maximummark(points) 8 12 12

A statistical analysis of peer assessment scores were performed for each individualcriteria as well as per essay (sum score). Analyses were performed using a randomcoefficientmodel subdividing the variance in the observed peer assessment scores intothe variance due to the performance of students, the variance due to the peers and theresidual error variance. Themodelwas used to assess the inter-rater agreement of thepeer assessments, agreement being defined as the degree towhich assessments on thesame student fromdifferent raters are consistent. Formally the agreement is defined asthecorrelationbetweentwopeersratingthesamestudentandisderivedfromthemodelastheratiobetweenthestudentvarianceandthetotalvariance(asinglemeasuresIntraClassCorrelation(ICC),rangefrom0to1).Valuescloseto0indicatenoagreement,valuesabove0.75excellentagreementandvaluesbetween0.4and0.75hasbeen suggested torepresentfairtogoodagreement(Fleissetal.,2003).Themodelcanalsobeusedtoderivetheagreementofaverageratherthanindividualpeerscores(theresidualerrorvarianceisscaleddownbykintheICCformulawhenanaveragescorebasedonkpeersisused(ICCaveragemeasures).Ascalecanbeusedtodiscriminatebetweenthestudentperformancesiftheresidualerrorvarianceissmallcomparedtothevariabilitybetweenthestudents.Toassessdiscrimination,anICCtypemeasurewasdefinedastheratiobetweenthestudentvarianceandthesumofthestudentandresidualerrorvariance.Toevaluatewhethertheself-assessment scoreswere systematicallydifferent from thepeerassessment scores inthe GEM course, the abovemodel was expanded with a fixed coefficient distinguishingbetweenself-andpeerassessment,therebyallowingforestimationofthemeandifferenceinscoresbetweenthetwokindsofassessments.

ResultsStudentproducedlearningelementsintheREEFcourseThe results of the questionnaire on the students’ evaluation of the student producedvideos are shown in Table 2. Overall students answered very positively to the


http://www.lom.dk 9

questions. On average 75% of the students strongly agreed or agreed to all sixquestions, 12.5% of the students answered neutral and 12.5% disagreed or stronglydisagreed.Morethan75%ofthestudentsagreedorstronglyagreedthatthemakingofvideosweremotivating,preferablecomparedwithhandinginareport,helpedthemtowork independently, stimulated creativity and should be learned by future students.56% of the students strongly agreed or agreed that making a video gave a betterunderstanding of the studied subject whereas 29% disagreed or strongly disagreed.Thisshowsthatwhereasasignificantmajorityofstudentsweregenerallysatisfiedwiththe application of student produced videos in the course, their perceptions of theireffects on learningoutcome is not unequivocally positive indicating that the learningoutcome fromapplying student produced videosdepends verymuchon the learningpreferencesoftheindividualstudent.

Table2.Student’sevaluationofstudentproducedvideos

Question Verymuch/stronglyagree

Tosomeextent/agree

Neutral Itdidnotmotivateme/disagree

Itde-motivatedme/stronglydisagree

1.Towhatextenthaveyouexperiencedtheuseofvideoasamotivatingfactorforyourworkwiththecourse?

37.5% 37.5% 12.5% 8.3% 4.2%

2.Inthiscourse,theuseofvideotodocumentandreportthefieldworkwaspreferablecomparedtohandinginaregularwrittenreport.

41.7% 41.7% 8.3% 8.3% 0%

3.Theuseofvideotocommunicatescientificworkontheinternetisapowerfultoolthatstudentsshouldlearnhowtomasterforuseintheirfuturecareer.

45.8% 29.2% 25% 0% 0%

4.Theprocessofmakingavideoinsteadofawrittenreportgavemeabetterunderstandingofthe

16.7% 41.7% 12.5% 25% 4.2%


http://www.lom.dk 10

studiedsubject.

5.ThevideosproducedinthiscoursehavegreatpotentialinhelpingrisingpublicawarenessaboutrestorationofecosystemsinEurope.

20.8% 50% 8.3% 20.8% 0%

6.Theproductionofavideohelpedmetoworkindependentlyandstimulatedmycreativity

41.7% 45.8% 8.3% 4.2% 0%

7.WhatdoyouthinkaboutusingYoutubeandotheronlinetoolsinuniversityteaching?

Mostlythestudentsansweredverypositivelytothisquestion.Studentsstatedthatthisisveryinterestingandusefulwayoflearningandteachingthatprovidedthemwithnewtooladaptedtothenewdevelopmentininformationsharing.Thesearejustfewofthestatementswegot:“Ithink,thatsoonitwillbeoneofthefundamentalwaysoflearning”.“Itwasveryunusuallyforme,butIthinktheonlinetoolsoffersomanynewpossibilities”.“ItisanadditionaltoolandagreatopportunitytoinvolveandinformpeoplewhicharenotconnectedtoUniversity.Writtenreportscontainmoredetailedinformationbutvideos/picturesaddressmoretheemotionsofpeopleandarethereforeprettypowerful”“Makingthefilmwasaninterestingexperience.Icouldlearnalotandhadalotoffun:)Itwasanicechangefromtheconstantsittingatadeskandreadingbooks,notesandarticles.Besidesimageshelpyoubettercommunicateandperpetuatethemverywellinmyhead”.

In thesubsequentclassroomdiscussionstudentsstatedthat theyappreciatedtheuseofvideoaslearningtoolandtheyfoundithighlyusefullearninghowtousevideointhedocumentationofscientificwork.Thestudentsexpressedthattheywerereallyhappywith the “movie nights”, that gave students the opportunity to collectivelywatch theworkoftheotherstudentsandgivedirectandinstantfeedbacktoeachother.Thiswasone of the most valuable experiences in the course both from the instructor's andstudent’s perspective. In connection with the embedded field work where samplingtechniques are deployed, students liked that the video gave them the opportunity toperformtheactualtechniqueinordertoshowtheminthevideoassignments.Studentsalso commentedon that the ability touse all available resources (voiceover, graphs,pictures,writtentext,music,animations,etc)providedgreat flexibility,butalsomadethe editing and uploading process more complex and time consuming. During theprocessofmaking thevideosabout their fieldwork(video2), studentsspenta lotoftime editing and uploading the videos. Thiswas exacerbated by the lack of previousknowledgewitheditingvideos, lackofcomputerswithenoughpowertohandle large



videofiles,weakinternetconnectionandtirednesssincestudentsdedicatetimetothevideoeditinganduploadingprocessintheafternoonafteralongdayoffieldwork.Thissituation was avoided with the last video (video 3), where students were asked tofollowtheDogmeconceptandtheydidnotuseanyotherresourcebutjustthefootagetaken on the field and spend much less time in editing and uploading.With respect to the use of the videos as tools for raising public awareness aboutrestorationprojects,studentsexpressedthattheydidnotfeelcomfortablewithhavingtheirvideosavailableonYoutube,howevermostofthemgaveuspermissiontousethevideoformarketingtheREEFcourseatacademicandpublicinstitutions.Animportantelementhereisthatmostofushaveanintrinsicfeartobejudgedbyother.TheactofmakingavideowithyourvoiceorimageinYoutubeoranyotherpublicplatformisahugechallengeformanypeople.

PeerreviewdiscussionsintheMPHseminarsTheresponseratefromquestionnaireswasaround50%=10students.Allthestudentsrecommended the seminars to continue, but the Danish students preferred autumn,whereastheEuropeanMasterofPublicHealthstudentspreferredspring:

“Ifeltlikethefirstsetofseminars,beforeChristmasbreak,wasnotthemostproductive.Joining the EUROPUBHEALTH and the Danish students at that point, and having them“compare” their projects was, in my opinion, counter-productive. The Danish studentswere a lot further ahead in their projects, and personally comparingmy progresswiththeirs just causedme anxiety. That is why I stopped attending. On the other hand, thepresentation of the projects in the spring terms was, I thought, very useful, and veryproductive, given thatweareall somuch closer to completingourprojects.However, Ithink that they should all be in the same period, so that the projects would be atcomparablestages.”

Whereas the students felt itwas abenefit toparticipate in the seminars, thereweretwohugebarriers.Oneofthemwaslanguage.ADanishstudentsaid:

“TalkingEnglishwasagreatbarrier toget100%of theseminar,and itwasdifficult torelate to the other international students work, especially because our deadlines weredifferent,( there isoneweekapart)andtheyquicklygotbehind,andweremorefocusedonthemethodsinsteadofthediscussion,whichweremorerelevantfortheMPH-studentstodiscuss.”

AnotherDanesaid:

“InmyopinionDanishandEUROPUBHEALTHstudentsshouldnothaveseminarstogether…..manygooddetailsandquestionsgetlost,whentheseminarsareinEnglish.”

However.WhereasmostoftheDanishstudentswereimpededbybeingtaughttogetherwithnon-Danishstudentssomeofthemfounditveryfruitfulfromaprofessionalaswellasaculturalpointofview.TheotherbarrierrelatedtopresentinginAdobeConnect.TheEUROPUBHEALTHeasily



feltfamiliarwithmakingapresentationatAdobe,whereassomeoftheDanishstudentswereafraidofusinganewtool.

“I have never seen anybody present research or health promotion by means of theinternet,sowhyshouldwespendourtimeonpreparingapresentation,especiallywhenwearesobusybypreparingourmasterthesis?”

The instructor´s evaluation of the seminars is thatEuropeanmaster of PublicHealthstudentsreallybenefittedfrompresentingintheseminars,maybebecausethatgroupof students was closely related and felt a kind of responsibility for each other andthereforecommentedon theprojects.TheDanishstudentswhopresentedwereverysatisfiedafterwards.Ittooksometimetopreparethispresentation,but“itwasfunny”.It was especially the younger Danish students who accepted this challenge.Thiswayofpresentingtopicshasbeenusedamongglobalhealthstudentswhoduringfieldtriponceaweekshouldpresenttheprogressof theirproposalstoclassmates inanothercountry.

ThisgroupofstudentsacceptedAdobeimmediatelywithoutquestionsandgotfruitfulfeed-back. Unfortunately, it was not possible to continue the seminars. The supportfrom the students was weak and since it was a 0 ECTS course it is difficult forinstructorstofindtimetoleadtheseminars.Theseminarscloseddownin2015.

PeerassessmentintheGEMcourse

ThenumberofparticipatingstudentsinaMOOCisnotwelldefined,butthesign-upsintheGEMcourseincreasedfrom1519to2736fromthebeginningtotheend.Atotalof199,170and138studentsdeliveredthethreeessayscalled1.Trend,2,Managementand3.TechnologyEssaysand138studentsdeliveredallthreeessays.ThedistributionofdeliverablesandselfandpeerassessmentscoresintheGEMcourseareshowninthetable:Table3.Distributionofdeliverablesandselfandpeerassessmentscores intheGEMcourse

Essay: Trend Management Technology

Deliverables 199 170 138

Selfassessments 175 149 125

Totalassessmentsinclself-assessments 754 647 522

Totalassessmentsexclself-assessments 579 498 397



Onestudentmadeexactly50peerassessmentsineachoftheessays,whichisfarhigherthananyoneelse.Toavoidplacingtoomuchemphasisononepersonintheanalyses,theassessmentsfromthispersonhavebeenexcludedinallanalyses(howeversimilarresultsareobtainedwhenkeepingtheseassessmentsintheanalyses).Moststudentsdidthe3requiredpeerassessments,andthereforemostofthedeliverableshavealsobeenassessedby3peers.Thedistributionofpeerassessments(excludingselfassessments)isshowninthebelowtable4:

Table4.Distributionofpeerassessments(excludingselfassessments)

Essay: Trend

(n=199)

Management(n=170)

Technology

(n=138)

Peerassessmentsperdelivery:

1-2 51 26 22

3 115 131 111

4-5 33 13 5

Assessmentsperformedperassessor:

1-2 1 1 0

3 148 130 108

4-10 28 23 16

Thenon-regulardistributionofnumberofassessmentsperdeliveryandperstudent(bothshouldbe3) aremainly results of a) Exclusionof the3*50 assessments by one student(about10%ofthedatamaterial),meaninglessassessmentsperdelivery,b)Thepersonalwishfromsomestudentstodoafewassessmentsmore,andb)Possiblylimitationsintheautomaticdistributionwhensomestudentschoosenottorateadeliverytheyreceive.Theresults from the statistical analysis is presented in Table 5. There are threeassignments/essayswithatotalofninecriteria.Theresultsareanalysedbothpercriteriaand per essay. Explanations of the headings are given in the text below.Table5.ResultsfromthestatisticalanalysisofpeerassessmentintheGEMcourse



Tr1 Tr2 M1 M2 M3 M4 Te1 Te2 Te3

Scoredistribution

1 6 84 10 23 33 51 6 8 54

2 82 163 118 215 230 172 28 20 50

3 323 286 519 409 384 424 124 112 164

4 343 221 364 382 254

Mean 3.33 2.85 2.79 2.6 2.54 2.58 3.63 3.66 3.18

Mean,%ofmax 83 71 93 87 85 86 91 92 80

Meanofsumscore 6.18 10.51

10.47

Meanofsumscore,%ofmax 77% 88% 87%

ICCdiscrimination 0.19 0.32 0.32 0.23 0.17 0.2 0.32 0.39 0.5

ICCdiscriminationsumscore 0.28 0.37 0.55

Pairwisedifferencesinpeerassessments,%

0 47% 40% 71% 54% 53% 51% 57% 60% 47%

1 45% 42% 28% 43% 42% 43% 36% 33% 37%

2 8% 16% 1% 3% 5% 7% 7% 7% 12%

3 0% 3% 0% 0% 4%

ICCsinglemeasure 0.16 0.27 0.27 0.18 0.15 0.16 0.26 0.29 0.41

ICCaveragemeasures,3assessors

0.36 0.52 0.53 0.40 0.35 0.37 0.51 0.56 0.68

ICCsumscoreaveragemeasures,singlemeasure

0.23 0.27 0.42

ICCsumscoreaveragemeasures,3assessors

0.48 0.55 0.68

No.assessorsneededforICCaveragemeasures>0.4

4 2 2 3 4 4 2 2 1

No.assessorsneededforICCsumscoreaveragemeasures>0.4

3 2 1



No.assessorsneededforICCaveragemeasures>0.75

16 9 8 14 17 16 9 7 5

No.assessorsneededforICCsumscoreaveragemeasures>0.75

10 8 5

Selfassessmenthighermeanscore

0.14 0.28 0.13 0.16 0.22 0.29 0.32 0.34 0.52

95%CI 0.02-0.25

0.13-0.43

0.05-0.20

0.06-0.25

0.11-0.32

0.18-0.40

0.21-0.44

0.23-0.46

0.35-0.69

Selfassessmenthigherscore,%ofmax.

5% 9% 7% 8% 11% 15% 11% 11% 17%

Selfassessmenthighermeanscore,sumscore

0.41 0.79 1.19

95%CI 0.19-0.64

0.51-1.06

0.89-1.49

Selfassessmenthigherscore%ofmax.,sums

7% 10% 13%

Themeansofthescoresaredividedbythemaximumpossiblescore,resultinginthe“%ofmax” tomake them comparable. The average scores for the criteria varies between71-93%and between 77-88% for the summary of the essays.No student ended below the60%averageofallthreeessays,thatwassetasthecut-offmark.Themarksarerelativelyhigh,butthatwasalsothe intentioninthisMOOCwherethestudentsdon’tobtainECTSpoints, that it ispossible topass ifyoudoa faireffort.Thosewhoactuallydidbother towrite an essay rather than just doing the quizzes could be expected to also do thenecessary effort. The ICCdiscrimination is not impressive as such, ranging from0.28 to0.55, supporting the observation that most marks are in the higher end.Theessentialresultsaretheotherhalfofthetablesummarizingthedifferencesbetweenassessors rating the same essays / students. The “Pairwise differences in peerassessments, %” shows the percentages of the differences between two assessors’assessmentofasinglecriteriathatare0(thesame),1,2or3.Morethanhalf(between40-71%) of the assessments made are precisely the same when two assessors look atparticularcriteriainanessay.28-45%ofthediffersby1scoremarkandmostlylessthan10% (1-19%) differ a score mark of 2 or 3.The ICC average measures show that with three assessors we will have a fair to goodreliability on assessments from the summarised (essay-based) results (ICC=0.48-0.68),whereason thesinglecriteria level (ICC=0.36-0.68) theassessmentsometimesrequirefour assessors. The results also show that on an assignment/sum score levelwewould



need 5-10 assessors for the assessment to have an “excellent” strength.Most of the students also did self assessment (89% of the deliverables) and whencomparing the scores given as self-assessment with the scores given by the peers thestudents tendtoscorethemselveshigherthantheirpeers,onaveragebetween0.13and0.52 higher scores corresponding to 5 to 17%higher than their peers, based on singlecriteria,orgenerally7to13%higherbasedonthecollectivescoringofeachessay.

DiscussionandconclusionBasedonamixedmethodsapproachfeaturingquestionnaires,focusgroupinterviewsandstatisticalanalysisdependingonthecoursebeinginvestigated,ourstudyhasgivensomevaluableinsightsintotheuseofstudentproducedlearningelements,peerreviewdiscussionsandpeerassessmentinauniversitysetting.

Byevaluatingtheuseofalearningdesignfeaturingstudentproducedlearningelementsinthe REEF course we have demonstrated that student produced videos is a viablealternative towritten group assignments be a good complement to thewritten text aspeer to peer learning elements and in the creation and communication of scientificinformation. Among other things students reported that working with videos weremotivating and stimulated both independence and creativity. This corresponds with astudy by Hafner andMiller (2011)who found that the use of student produced videoswere highly motivating and increased independent learning. Whereas most studentsgenerallyperceivedtheuseofvideosasagoodthing,theydidnotallagreethattheyweresuperior to a written report with respect to learning outcome. This contrasts with thestudiesbySildis(2006)whofoundthatvideoprojectsmayincreaselanguagelearningandHoogerheide et al (2016) who found that explaining on video was more efficient forlearning than explaining in text. The production of videos by students that have beeneducatedinatextbasededucationsystemimposedachallengewhichrequiredstudentstoinvestalotoftimeandeffortinmasteringvideoeditinganduploadingskills,whichmayhavetakentimeawayfromachievingthecorescientific learningoutcome.However, inastudy comparing the quality of an assignment submitted as a video podcast or on a CDÖzdenerandGüngor (2010) found that therewasnodifference in the learningoutcomeandacademicquality, thequalityof thedesignandpresentationwashigher inthevideopodcast thanontheCD.Theuseofvideospublishedonthe internethas thepotential toraisepublicawarenessbeyondtheclassroom,butourstudyshowsthatitisimportanttoprovidepropertimeandguidanceforit.

Even though several universities and international education programmes offer onlinethesisseminarswithpeerfeedbacktostudentseitherasavoluntaryormandatorypartoftheirstudies,(e.gAaltoUniversityandCrossBorderUniversityinFinland,andUniversityofAmsterdamandLeidenUniversityintheNetherlands)wehavenotbeenabletofindanyother studies describing the evaluation of such online thesis seminars. In the MPHseminars it was evident that students who took time to present and comment onpresentations by other students really benefitted from the time they spend on thislearning activity. Speaking English as well as technical issues were the main barriers



among the Danish students. Since the students learn so much it is recommended thatlearningdesignsthatintegratethesisseminarsandpeerfeedbackarereintroducedintheMPH as a mandatory requirement before the thesis can be submitted.Withrespect to theuseofpeerassessmentourstudyof theGEMcoursehasshownthatthereisareasonablegoodagreementbetweenassessorsindeterminingtheaveragescorefor multiple essays, requiring only three assessors to give a reliable average score,whereas 5 to 10 assessors are required to give reliable scores for individual essays. Itfollowsthatcourseswithalimitednumberofpeerassesseddeliverablesmustsecurethateach deliverable is assessed by a sufficiently large number of peers to validate theassessment. However, as argued by Suen (2014) there is a risk that unmoderated peerassessment may suffer from lack of credibility. In a MOOC course on human centeredinteraction design Kulkarni et al (2013) compared student assessments with teacherassessmentsandfoundthatthateventhoughthemajorityofstudentsassessedtheirpeerswithin 10% of the teacher assessments, students generally assessed their peers higherthan the teachers (7%). Furthermore, they also assessed peers from their own countryhigher thanpeers fromothercountries. Ithasbeensuggestedthat thereliabilityofpeerassessment in MOOCs may be improved by guidance and training (Ashton and Davies,2015). Nevertheless, peer assessment has it’s limitations and (Admiraal et al, 2015)suggests that in the context of MOOCs self-assessment and peer assessment should beusedasassessmentforlearningandnotassessmentoflearning.

Byinvestigatingdifferentapproachestotechnologyenhancedpeerlearningourstudyhasshownthatstudentproducedvideosmaybeusedasanalternativetowrittenassignments,but thatproperguidanceand instruction isneeded tosecure that studentshaveenoughtimetoachieve thedesired learningoutcome.Whereas theadvantagesofpeerreviewedonlinethesisseminarsseemsstraightforward,properguidanceandtrainingisalsoneededto improve the quality and learning outcome of online peer assessment. An overallconclusion from our study is that the application of learning designs featuring studentproduced videos, peer reviewed online thesis seminars and online peer assessment arevaluableadditionstothemoretraditionalinstructordrivenlearningandassessment,butthat teacher guidance to facilitate formal peer learning is very important. It should benotedthatthisconclusionisbasedonthemixedmethodsapproachappliedinthestudy,and that amore quantitative evaluation of student produced videos and peer reviewedonline thesis seminars and amore qualitative evaluation of peer assessmentsmay giveother results. Furthermore, it is suggested that future studies should also explore howinformal peer learning could be facilitated to reinforce knowledge sharing practices inonlineandblendedlearningcourses.



ReferencesAdmiraal,W.,Huisman,B.,Pilli,O.(2015).AssessmentinMassiveOpenOnlineCourses.ElectronicJournalofE-learning13,pp207-216

Airasian,P.W.,&Russell,M.K.(2001).Classroomassessment:Conceptsandapplications.Boston:McGraw-Hill.

Ashton,S.,Davies,R.S.(2015).UsingscaffoldedrubricstoimprovepeerassessmentinaMOOCwritingcourse.DistanceEducation36,pp312-334

Bruner,J.S.(1986).ActualMinds,PossibleWorlds.HarvardUniversityPress,Massachusetts,USA

Burnett,S.,Blakemore,S-J.(2009).Thedevelopmentofadolescentsocialcognition.AnnalsoftheNewYorkAcademyofScience1167,pp51-56

Dochy,F.J.R.C.,Segers,M.,&Sluijsmans,D.(1999).Theuseofself-,peerandco-assessmentinhighereducation:Areview.StudiesinHighereducation,24(3),331-350.

Fleiss,J.L.,Bruce,L.,&MyungheeC.P.(2003).Statisticalmethodsforratesandproportions.JohnWiley&Sons.

Hafner,C.A.,Miller,L.(2011).FosteringlearnerautonomyinEnglishforscience:Acollaborativedigitalvideoprojectinatechnologicallearningenvironment.LanguageLearning&Technology15,pp68-86

Hakkarainen,P.,Saarelainen,T.,&Ruokamo,H.(2007).Towardsmeaningfullearningthroughdigitalvideosupported,casebasedteaching.AustralasianJournalofEducationalTechnology,23(1),87.

Hoogerheide,V.,Deijkers,L.,Loyens,S.M.M.,Heijltjes,A.,vanGog,T.(2016).Gainingfromexplaining:Learningimprovesfromexplainingtofictitiousothersonvideo,notfromwritingtothem.ContemporaryEducationalPsychology44-45,pp95-106

Kulkarni,C.,Wei,K.P.,Le,H.,Chia,D.,Papadopoulos,K.,Cheng,J.,Koller,D.,Klemmer,S.R.(2013).PeerandSelfAssessmentinMassiveOnlineClasses.ACMTransactionsonComputer-HumanInteraction20,33

Lave,J.&Wenger,E.(1991).Situatedlearning:Legitimateperipheralparticipation.CambridgeUniversityPress.

Nehls,N.(1995).Narrativepedagogy:rethinkingnursingeducation.JournalofNursingEducation34,204–210

Piaget,J.(1932).TheMoralJudgementoftheChild.NewYork:TheFreePress.

Roberts,D.(2010).Vicariouslearning:areviewoftheliterature.NurseEducationinPractice10,pp13-16

Sildis,T.I.(2006).TheEffectofaStudentVideoProjectonVocabularyRetentionofFirst-YearSecondarySchoolGermanStudents.ForeignLanguageAnnals39,pp54-70



vanZundert,M.,Sluijsmans,D.,vanMerriënboer,J.(2010).Effectivepeerassessmentprocesses:Researchfindingsandfuturedirections.LearningandInstruction,20,270-279.

Vygotsky,L.S.(1978).Mindinsociety:Thedevelopmentofhigherpsychologicalprocesses(A.R.Luria,M.Lopez-Morillas&M.Cole[withJ.V.Wertsch],Trans.)Cambridge,Mass.:HarvardUniversityPress.(Originalwork[ca.1930-1934])

Wenger,E.(1998).CommunitiesofPractice:Learning,Meaning,Identity.CambridgeUniversityPress

Özdener,G.andGüngor,Y.(2010).Effectsofvideopodcasttechnologyonpeerlearningandprojectquality.Procedia-SocialandBehavioralSciences2,2217-2221

ku · learning elements, peer review discussions and peer assessment in the bsc/msc level summer...

Documents