D E S I G N I N G P E E R F O R M A T I V E A S S E S S M E N T

W I T H O P E N B A D G E S I N O N L I N E L E A R N I N G E N V I R O N M E N T S

S T Y L I A N O S H AT Z I PA N A G O S , K I N G ’ S C O L L E G E L O N D O N & S T E V E N W A R B U R T O N , U N I V E R S I T Y O F S U R R E Y

CONTEXTA project that explored the effectiveness of peer feedback and assessment; Investigated whether open badges can support learning in online environments through enhanced engagement; Revisited and evaluated badge typologies with the aim of identifying the most appropriate typology that supports peer feedback and assessment; And created a clear correspondence between the typology of badges and peer feedback and assessment practice in online classrooms. 

RATIONALE

Use of digital badges to support peer feedback and assessment in higher education

By the use of open badges in encouraging and rewarding competences, knowledge and skills acquired in digital learning spaces, we aimed to reinforce the value of peer assessment and by extension peer feedback and review activities in supporting student learning.

PEER ASSESSMENT DEFINITIONPeer assessment is to be understood as an educational arrangement in which students assess the quality of their fellow students’ work and provide one another with feedback.

Dochy et al., 1999

FORMATIVE E-ASSESSMENT CAN: • engage learners with tasks that enhance learning

outcomes and test the development of higher order capabilities;• enrich learning approaches by making feedback

central to all assessment activities;• promote a dialogue in relation to feedback, peer and

self assessment activities.Hatzipanagos & Warburton 2009

PEER ASSESSMENT/PEER REVIEW• Peer assessment is frequently underused in online

learning environments. • Its informal learning value is not exploited by peers as

it is not explicitly linked to summative assessment activities. • There is a reluctance to participate in such peer

review activities and an ambivalence of tutors in relation to their value. • Informal aspect is not clearly or meaningful

embedded in the learning process.

Peer assessment attributes (Dochy et al)

What the research says

Badges

Validity Peers are prone to produce ratings based on uniformity, race and friendship if there is no extensive training in peer rating.Majority of studies showed an acceptably high validity and reliability in a varietyof fields.

Clear criteria embedded in badges definition. Validity can be high if badges are well defined.

Fairness. Student perceptions of fairness of peer assessmentStudents were very positive about the effects of alternative assessment on their learning.

Openness and clarity in microcertification fundamental requirements of a fair and valid assessmentsystem.

Accuracy. Agreement between student and tutors. No consistent results.

There can be ‘objective’ approaches of testing accuracy in microcertification

Effects. Peer assessment fostered an appreciation for the individuals' performance within the group and interpersonal relationships in the classroom

Badges can capture interpersonal relationships (soft skills) and performance (hard skills)

Implications for practice.

valuable as a formative assessment method but… friendship marking; collusive marking; decibelmarking; and parasite marking..

Appropriateness of emergent technologies such as badges for formative assessment Assessing ‘mishaps’ can be prevented by learning analytics and digital trail in new environments

METHODOLOGYThe Open Badges used in King’s College London and the University of Surrey in the UK in two distinct programmes. King’s programme (Postgraduate Certificate in Academic Practice) targets novice lecturers, PhD students with a teaching role, librarians and information specialists who have chosen to study a ‘Supporting technology enhanced learning’ module. Peer Assessment is an integral part of these modules and learners are strongly encouraged to participate in formative assessment activities. There was an opportunity to link performance and peer review in these activities with open badges.

METRICS• Assigning badges was a semi-automated process. • Use of the learning environment triggered automated assignment of badges for metrics

associated with use of the learning environment e.g. in the context of computer mediated communication, learning behaviours were monitored and examined when students used discussion fora and the social media communication. Metrics such as number of posts to a discussion forum, number of log-ins, response to other posts, sharing content and reflections were used.

• Other badges were deployed by application and evidence of achievement, moderated either by the module tutors, the project team or by peers on the course. A protocol for awarding a badge was developed by the project team in collaboration with the module tutors and learners e.g. we specified the group of users who could approve or reject badge awards.  

• Evaluation of the project activities was carried out by collecting data from the online learning environment, using focus group interviews and by end of module and programme evaluation questionnaires.

PROJECTIONS• By rewarding positive behaviours the participants would feel

more motivated to complete learning activities. The aim of this approach was to manage any motivational drops that can commonly be experienced in online/blended courses.• Participants were encouraged to push their practice beyond the

central learning design journey and engage in the more challenging aspects of the activities provided. For example, in terms of their engagement with a wider community, and development of their identity and responsibilities within, across and beyond that community.• The project would provide a mechanism for self-reflection by

the academics engaged in the study. They would be able to evaluate the benefit of using a badging system within their own course designs.

CONCRETE EXAMPLES

• For example, the ability to award a badge to your peer evaluator if they provide good constructive advice on your work.

Powered by

Digital Badges evaluation

Q2: HOW DO BADGES COMPARE WITH OTHER WAYS OF DEMONSTRATING YOUR ACHIEVEMENTS?

Q5: DO YOU THINK THAT BADGES MOTIVATE LEARNERS IN AN ONLINE ENVIRONMENT?

Q14: DO YOU SEE BADGES BECOMING PART OF YOUR DIGITAL ACADEMIC IDENTITY?

PERCEPTIONS OF VALUE• Random orchestrations between learners that are not part of the same

community. ‘Due to the limited time I spent with my fellow students I didn't get to know them well. Therefore I felt that the use of badges was not particularly useful’• Soft indicator of achievement, used for formative rather than summative

purposes. ‘in lieu of marking or registers they could be used as a 'soft' indicator of achievement (e.g. a record of optional on-line courses taken)’ • Overall caution against the use of badges ‘don't think the possibility of winning a digital badge really changes adult behaviour’‘unless digital badges are established to international- or the very least national standards as a well recognized metric of achievement I wouldn't think they will affect online learning.’• Student centred approach should work‘something that the students themselves defined and awarded, rather than it coming from me (also a good way to gauge what they think of the whole thing!)’

WHAT DO YOU THINK WOULD HELP THE ADOPTION OF BADGES? 

• I think a clear rationale for the benefits to both the student and staff • Defining their purpose more clearly, and applying them consistently to something that is meaningful to students' learning

and achievements. Instinctively, it's quite hard to take them seriously. Is it the name? or the varied application? Not sure. • If someone badges feed into the assessment process -i.e. maybe 5 extra marks could be added to the summative

assessment to recognize participation and contribution. • If they were linked to module credit • As the criteria for digital badges are still very loose I don't think it is likely that they are used in evaluating academic

contribution. Stricter and well defined metrics are in place for academic professional development (at least in Life Sciences). 

• I might be wrong about this - and it's a bit of a chicken and egg situation - but I don’t see them as an important part of a professional identity, and I gather from friend on corporate e industry (e.g. cloud computing) that they are not taken seriously there either. If I felt that they were something that employment panels cared about and that validated my use of e-learning tools, then I'd swallow my scepticism, adopt them enthusiastically and show off about the ones I have earned. Of course, if none of us decide to adopt that approach then the panels won't care, and so on around the circle... it's just really difficult for me not to see them as something rather childish and believe that my students would feel the same.

• I think there's potential use for badges as part of the summative assessment (which would then require further guidelines and transparency).

• Academia is already a game measured by the 'badges' of qualifications, publications, citations and research grants -- further gamification seems unnecessary

WHAT WE WOULD LIKE TO SEE IN THE FUTURE • Integration of badges with university systems; value

of badges for summative assessment; promotion; recruitment• Better integration of badges with external certification

systems (Mozilla open badges, Open Badge Factory) and these with social media• Clear distinction between ‘hard’ and soft ‘transversal’

skills; but both essential in peer assessment activities• Embed micro-portfolios of badges in PDP (personal

development planning) and portfolio building for staff and students.

REFERENCES

• Dochy , F. , Segers, M. & Sluijsmans, D. (1999) The use of self-, peer and co-assessment in higher education: A review, Studies in Higher Education, 24:3, 331-350, DOI:10.1080/03075079912331379935. 

• Hatzipanagos, S. & Warburton, S. (Eds) (2009). Social Software and Developing Community Ontologies. 598 pp. London: Information Science Reference, an imprint of IGI Global ISBN 978-1-60566-208-4.

IF YOU HAVE QUESTIONS OR FEEDBACKEmail s.hatzipanagos@kcl.ac.uk