september 2007 practice in e-assessment conference increasing learner success with technology...

September 2007 Practice in e-Assessment Conference

Increasing learner success with Increasing learner success with technology supported assessment: technology supported assessment:

findings from the REAP projectfindings from the REAP project

Catherine Owen, Project Manager

www.reap.ac.uk

September 2007Practice in e-Assessment Conference

The REAP ProjectThe REAP Project

• 3 HEIs (Strathclyde, Glasgow Caledonian Business School, Glasgow University)

• Focus is on large 1st year classes• Pedagogy: assessment for learner self-regulation• Range of technologies: online tests, simulations,

discussion boards, e-voting, e-portfolios, peer/feedback software, admin systems, VLEs, offline-online

• Goals: Learning quality and teaching efficiencies• Outputs: case studies of redesign: advice to support

strategic change in institutions (transformation).• Dissemination across HE/FE sector


PlanPlan

• Review literature on formative assessment and feedback• Suggest some simple ways of improving assessment

and feedback practices in relation to the research• Outline some interesting case studies of first year course

redesign drawn from REAP project and elsewhere.• Analyse these case studies in relation to assessment

principles from literature• Questions and discussion


Defining assessment and feedbackDefining assessment and feedback

• Assessment – the narrow meaning is exam for external accreditation

• Feedback – the narrow sense is what the tutor writes about/on a finished piece of student work

• But in this talk also interested in:• Self-assessment and reflection - a form of internally

generated feedback• Peer dialogue as feedback and peer assessment


Definitions (2)Definitions (2)

• Who is involved in formative assessment and feedback

• Tutor• Peers• External (e.g. placement supervisor)• Computer generated• Self


Why take formative assessment and feedback seriously?Why take formative assessment and feedback seriously?

• Assessment is a key driver of student learning• Assessment is a major cost in HE (staff time)• Widely reported that students don’t read the feedback

• Dropout/retention – linked to academic experience• First year experience – students need regular and

structured feedback opportunities.• National student survey – students are dissatisfied with

feedback.• QAA reports – main area of criticism in England


Assessment paradigm

Transmission

[teacher-centred]

Teaching/learning paradigm

Transmission

Research on Assessment in HEResearch on Assessment in HE

Constructivist [student-centred]


Key messagesKey messages

• Formative assessment and feedback by others can only have an impact on learning when it influences a student’s own self-regulatory processes (adapted from Boud, 1995)

• Students are already self-assessing and generating feedback so we should build on this capacity (Nicol and Macfarlane-Dick, 2004)


Scaffolding self regulation: 7 principles of Scaffolding self regulation: 7 principles of good feedback (assessment design)good feedback (assessment design)

1. Clarify what good performance is (goals, criteria, standards).

2. Facilitate reflection and self-assessment in learning

3. Deliver high quality feedback to students: feedback that enables students to monitor and self-correct

4. Encourage peer and tutor dialogue around learning

5. Encourage positive motivational beliefs & self esteem through assessment

6. Provide opportunities to close the feedback loop

7. Use feedback information to shape teachingSource: Nicol and Macfarlane-Dick (2006),


1. Help clarify what good performance is1. Help clarify what good performance is

• Documents with printed criteria/ published online• Tutor explains/discusses criteria with students• Frequently asked questions database (FAQs)• Exemplars of performance with feedback (e.g. essays -

- made available online)• Students discuss good and bad examples of

assignments and identify criteria• Students rephrase criteria in own words• Students generate their own criteria for a task• Students construct own MCQs and post online


2. Facilitates self assessment 2. Facilitates self assessment and reflection in learningand reflection in learning

• Online quizzes (e.g. MCQs/short answers) with feedback (self monitoring)

• Self-assessment at submission (strengths/weaknesses)• Open book use of online MCQs• Peer processes where students judge work of others

against standards and give each other feedback• Simulations – dynamic feedback shows effects of actions• PDP involving ongoing reflection on learning (e-

portfolios)• Confidence ratings with MCQs increases reflection


3. Deliver high quality feedback information: 3. Deliver high quality feedback information: that enables students to self-correctthat enables students to self-correct

• Link feedback to pre-defined criteria (comment databanks)

• Provide feedback soon after submission (clarify expectations)

• Point to resources where answers can be found (hyper-links)

• Have students request feedback

• Reader-response theory and feedback


4. Encourage teacher and student 4. Encourage teacher and student dialogue around learningdialogue around learning

• Break out discussions in tutorial to discuss written feedback and plan strategies for improvement

• Students give descriptive feedback to peers before submission

• Electronic voting technology – three levels of feedback (computer, peer and teacher)

• Unique weekly assessed tutorial sheets

• Online discussion as peer feedback (psychology example)


5. Encourage positive motivational beliefs 5. Encourage positive motivational beliefs (self-efficacy) and self-esteem(self-efficacy) and self-esteem

• Online objective tests (repeatable, private, comparisons with own learning goals)

• Using authentic ‘real life’ tasks that require the kinds of thinking and skills required in employment

• Structured milestones with early low stakes tasks and feedback

• Focus on learning goals not just performance goals (e.g. rewarding effort and improvement)

• Two-stage assignments (drafts-feedback-marks)

• Supporting formation of learning communities around tasks (discussion boards)


6. Provide opportunities to close 6. Provide opportunities to close the feedback loopthe feedback loop

• Increase opportunities for resubmission

• Two-stage assignments (drafts-feedback-marks)

• Action points as feedback strategy

• Personal development planning – students keep records of work at different stages

• Initiate online peer review processes as available to academics when writing journal articles

• Not releasing the grade until students have commented on the feedback provided.


7. Provides information that can be used to 7. Provides information that can be used to shape the teachingshape the teaching

• Student requested feedback (questions worth asking)

• Electronic voting systems

• Just-in time-teaching using MCQs

• Monitoring online discussions

• Monitoring project work within shared workspaces

• Blackboard/WebCT have some built in reporting functionality about student activity


DiscussionDiscussion

• Would any of this apply to your situation? How?• How would you improve on the principles

described?• Why don’t these ideas apply?• What would you do instead?• Any questions?


Two super principlesTwo super principles

Super-principle 1: developing learner self-regulation (empowerment) i.e steers to encourage ownership of learning – the seven principles discussed above.

Super-principle 2: time on task and effort (engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditions

Case examples from REAP – applying these principles/conditions


Case 1: Case 1: Using a VLE to support collaborative Using a VLE to support collaborative

group tasksgroup tasks(University of Strathclyde Psychology (University of Strathclyde Psychology

Department)Department)


Case 1: PsychologyCase 1: Psychology Context:

• 560 first year students

• Mixture of psychology majors (130) and those taking psychology only for one year (430)

• 6 topic areas, 48 lectures, 4 tutorials, 12 practicals

• Assessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in experiment (5%), essay exam (66%)

• 85 VLE groups of 7-8 students

Aims:

• To develop essay writing skills and provide effective feedback

• To reduce lectures and burden on staff


Structure of assigned group tasksStructure of assigned group tasks 6 cycles of 3 weeks (one per major course topic)

Each cycle:

• Week 1: "light" written task, ≈ 7 short answers, between group members.

• Week 2: Reading task only

• Week 3: "Heavy" written task, typically 7 short answers followed by combining them into a single coherent essay.

Within each week is:

• The Monday lecture, introducing material.

• Immediately afterwards, that week's task posted online, for delivery before next Monday.

• Model answers (a few selected from the best student answers submitted) are posted for the previous week's task.


Big success, apparently!Big success, apparently!

• Students set targets beyond what was required: contradicts commonly held beliefs about assessment and external motivation (marks)

• Produced work at level ‘not seen before’ surpassing third year• Spontaneous discussions about learning and learner responsibility• Some students burdened by workload but easily detected• Some groups participants moved at own request (3 groups)• 13,429 messages posted by groups (postings from 40-400 per group)• Quality of interactions ‘outstanding’ across the board • Atmosphere in lecture class improved and online community developed


Case 1 and the principlesCase 1 and the principles

• Model answers and repeated task format provide progressive clarification of expectations (clear criteria, A1)

• Students encouraged to self-assess against model answer (A2)

• Online peer discussion aimed at reaching agreed response (A4)

• Staged complexity and focus on learning rather marks (A5)

• Repeated cycle of topics and tasks (A6)*

• VLE captures all group interchanges, allowing course leader to monitor progress and adapt (A7)

• Tasks require significant study out of class (capture enough work B1)*

• They are distributed across topics and weeks (are spread out evenly B2)*

• They move students progressively to deeper levels of understanding (B3)

• There are explicit goals and progressive increase in challenge (communicates clear and high expectations B4)


Case 2: Case 2: Using a VLE to support collaborative Using a VLE to support collaborative

group tasksgroup tasks(Glasgow University Biology Department)(Glasgow University Biology Department)


Biology CaseBiology Case

• 700 students in first year class

• Module core for biologists but also optional for numerous other subjects

• Group task a key element of the class

• Encouraging communication and group working skills, independent research and presentation skills



• Students work in groups to create a poster investigating a species (chosen by the group) and its environmental impact

• Each group must argue for the extinction of a species in a head to head debate with another group in class

• An overall group mark is given by the tutor (for poster and debate), then peers moderate the marks in their groups to reflect individual contributions

• Participants also asked to reflect on their own contribution to the group and write a paragraph explaining how they added value to the task



• Improve the students’ learning experience (with positive impact on retention and progression)

• Support students to participate in the group discussions regardless of the timing or other commitments (family, travel, work etc.)

• Avoid problems arising from student absence and ‘no shows’, since all students in a group will have access to material ‘posted’ in the forum

• Reduce tutor mediation time by providing a permanent record of the group interactions for both students and staff



• 80 web forums for 80 groups of 8 members created on Moodle

• Group tasks released by tutors as a series of progressive mini-tasks with short submission deadlines (e.g. rationale for species choice)

• Tutor feedback provided to whole class via open message-board

• Peer-marking process refined to include pre-task group definition of criteria and a post-task reflection on and revision of criteria (reflection on criteria)


Benefits of the redesignBenefits of the redesign

• Using the VLE forums was popular with students• Positive impact on student confidence and the

development of face-to-face as well as online interactions

• Peer feedback valued as an addition to tutor feedback• Students re-assured that contributions monitored by staff

to support fair peer marking process• Tutor mediation in group disputes significantly reduced• Anecdotal evidence of improved class cohesion and

greater social interaction around learning• Quantitative results pending…


Relation to seven feedback principlesRelation to seven feedback principles

• Group discussion around criteria for marking helps to define goals and standards (principle 1, clear goals)

• Peer marking helps students to develop objectivity in judgement. Students self-assess their own contribution to the group and give a rationale for their contribution. (principle 2, self-assessment)

• GTAs monitor contributions of group members but teacher provides generic feedback to the open discussion board (principle 3, teacher feedback)

• Peer dialogue and feedback happens in groups and in producing a group submission to forum. Online interactions support the development of learning communities online and face-to-face (principle 4, peer dialogue)

• Opportunity to communicate with peers online as well as F2F liberating for many shy or anxious students (principle 5)

• Discussion boards provides record of all student outputs which teachers can use to make adjustments in how they teach (principle 7, shape teaching)


Relation to Gibbs and Simpson’s four conditionsRelation to Gibbs and Simpson’s four conditions

• Structured, progressive tasks encourage time and effort on tasks (condition 1, captures sufficient study time and effort)

• Requires students to work in and out of class and make regular contributions (condition 2, distribute student effort)

• Debating with other students encourages deep rather than surface learning (condition 3, productive learning)

• Marking criteria are negotiated and the tasks are challenging (condition 4, clear and high expectations)

• Not forgetting efficiency gains in course delivery...


Case 3: Case 3: Podcasts, electronic voting and SATsPodcasts, electronic voting and SATs(University of Strathclyde Hospitality and (University of Strathclyde Hospitality and

Tourism Department)Tourism Department)


Case 3: Tourism and ManagementCase 3: Tourism and Management

• 230 students, level course, two semesters

• Traditionally been delivered as two lectures per week supported by four one-hour tutorials per semester

• Attendance at lectures poor and declines a lot throughout the year

• Changes: lectures halved and replaced by:

• Interactive lectures using EVS

• Regular online SATs replacing class tests

• Video-podcasts of course materials

• Group presentations supported by online micro-tasks


Case 3: Tourism and ManagementCase 3: Tourism and Management

• Attendance at lectures improved

• Number of non-qualifying students reduced (from 40 to 30)

• Average coursework mark rose from 47.8% to 60%


Case 3 and the principlesCase 3 and the principles

• Online SATs, EVS allow self-assess against given answer, and the class’ response (A2)

• Online SATs & EVS usually followed by teacher’s model answer (A3)

• Group presentation encourages peer dialogue around task(A4)

• Success at tests (rather than uncertainty) promotes self-esteem (A5)

• Repeated cycle of tasks gives opportunities to close feedback loop(A6)

• EVS also lets teacher assess state of the class as a whole (A7)

• Podcasts, texts to be used before class (capture enough work B1)

• They are distributed across topics and weeks (are spread out evenly B2)


• Involve learners in decision-making about assessment policy and strategy

• Give choice in assessment -content and processes

• Facilitate reflection and self-assessment in learning

• Encourage positive motivational beliefs & self-esteem

• Deliver quality feedback that helps learners self-correct

• Encourage time & effort on purposeful learning tasks

• Help clarify what good performance is

EMPOWERMENT(Adapt institution to student)

SOCIALEXPERIENCE• Encourage interaction and

dialogue around learning (peer and teacher-student)

ENGAGEMENT(Fit student to Institution)

• Support the development of learning communities

Figure 1: A framework and ten principles for formative assessment and feedback in the first year and beyond

• Adapt teaching to student needs

ACADEMICEXPERIENCE

© d.j.nicol 2007


Relevant papersRelevant papers• Nicol, D (in press), Laying the foundation for lifelong learning: cases studies of technology

supported assessment processes in large first year classes, British Journal of Educational Technology (to be published July 2007).

• Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education.

• Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge.

• Nicol, D (2006), Increasing success in first year courses: assessment redesign, self-regulation and learning technologies, Paper prepared for ASCILITE conference, Sydney, Australia, Dec 3-6.

• Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199-218.

• Nicol, D & Boyle, J. (2003), Peer Instruction versus Class-wide discussion in large classes. Studies in Higher Education, 28(4), 457-473

• Boyle, J.T. and Nicol, D. J. (2003). Using classroom communication systems to support interaction and discussion in large class settings, Association for Learning Technology Journal, 11(3), 43-57