Motivating students to engage with assessment Motivating students to engage with assessment and self-regulate their own learning

Ali Wyllie and Ingrid Nix Lecturers in Learning and Teaching Faculty of Health and Social Care

HEA e-learning for health Conference

Motivating students to engage with assessment

1

Motivating students to engage with assessment regulate their own learning

learning for health Conference –S3BP6 (17 July 2009)

Research context

• Competing priorities of work and study mean HSC students selectively engage in computer

• HSC students typically low in confidence in ICT

• How can we: – increase their motivation– enhance self-regulation (ability to manage own learning).

Competing priorities of work and study mean HSC students selectively engage in computer-based assessment

HSC students typically low in confidence in ICT

2

regulation (ability to manage own learning).

Learning with Interactive Assessment (LINA) project Our research questions:

• What features would enhance learner’s selfand motivation to engage?

• Would a confidence indicator tool and a learning log promote a more reflective approach to learning and formative assessment?

Learning with Interactive Assessment (LINA) project

What features would enhance learner’s self-regulation

3

Would a confidence indicator tool and a learning log promote a more reflective approach to learning and

Confidence

LINA research areas

Does this encourage reflection and evidence-based practice?

Learning log

Other - Feedback

Confidence indicator

Does this encourage reflection on choices made & input of additional thoughts?

In what ways do students draw on Feedback?

practice?Media

LINA research areas

Does variety of media make Qs more motivating (e.g.

audio, animations)?

Does narrative within

4

resources

Sequence of

resourcesOther -

Feedback

Does narrative within sequences make a difference? Does choice make a difference, selecting easier or more difficult (e.g. to make more effective use of study time)?

LINA website

Animation window

5

Re-engineering Assessment Practices‘Good feedback practice is anything that might strengthen the

students’ capacity to self-regulate their own performance

REAP identifies 7 principles of good feedback practice,REAP identifies 7 principles of good feedback practice,1. helps clarify what good performance is (goals, criteria, expected

standards)2. facilitates the development of self

learning3. delivers high quality information to students about their learning4. encourages teacher and peer dialogue around learning5. encourages positive motivational beliefs and self6. provides opportunities to close the gap between current and desired

performance7. provides information to teachers that can be used to help shape teaching.

engineering Assessment PracticesGood feedback practice is anything that might strengthen the

regulate their own performance’Nicol and Macfarlane-Dick (2006:205)

REAP identifies 7 principles of good feedback practice, as feedback which:

6

REAP identifies 7 principles of good feedback practice, as feedback which:1. helps clarify what good performance is (goals, criteria, expected

2. facilitates the development of self-assessment (reflection) in

3. delivers high quality information to students about their learning4. encourages teacher and peer dialogue around learning5. encourages positive motivational beliefs and self-esteem6. provides opportunities to close the gap between current and desired

7. provides information to teachers that can be used to help shape teaching.

Method – research instruments

1. LINA website introducing two topics of 10 questions. 2. Feedback questionnaire.3. Video of 3 participants using LINA in a ‘Userlab’ facility

using a Think-Aloud protocol.4. Learning log of participant’s actions and reflections. 5. Researchers observation notes taken during videoing.6. Transcripts of audio interviews following Userlab testing.

research instruments

1. LINA website introducing two topics of 10 questions.

3. Video of 3 participants using LINA in a ‘Userlab’ facility

7

Aloud protocol.4. Learning log of participant’s actions and reflections. 5. Researchers observation notes taken during videoing.6. Transcripts of audio interviews following Userlab testing.

Method - sample

2007Invitation K216 Course

website message to website message to 460 students

Respondents completed Topics and submitted feedback questionnaire

5 students2 colleagues

IET Userlab testers 3 of the aboveTotal 7

* IET - Institute of Educational Technology

2008K216 Course website message to

IET email to 255 K216 students

8

website message to 460 students

K216 students

5 students2 colleagues

5 students

3 of the above n/a5

Institute of Educational Technology

LINA FindingsLINA Findings

9

Confidence-indicator toolEngage with

question

How my mark is

calculated

Select my confidence

level

indicator toolEngage with

questionType my answer

10

Select my confidence

level

Reflection, Justification, & Score• ‘…it makes you think about what

you’ve just said… It makes you evaluate... how well you think you’ve done’. [D-78]

‘…if I saw little gold

Important as a prompt to reflection and

justification of learner choice (Principle 2).

‘…if I saw little gold stars or something…I think that would motivate me

Reflection, Justification, & Score‘…it makes you think about what you’ve just said… It makes you evaluate... how well you think

‘‘…it just introduces a bit of an edge to it in a gambling sort of sense’. [C-98]

if I saw little gold

11

Score-value important for encouragement and motivation (Principle 5).

if I saw little gold stars or something…I think that would motivate me’ [S-188].

Learners and self-• ‘… when you’re highly

confident and then you don’t get it right, that inevitably makes you stop and think’. [C-104]

Achievement-oriented (score-driven) strategy still leads to reflection

(Principle 2)

-assessmentLearning-oriented approach encourages reflection (Principle 5).

12

‘It wasn’t so much about the score, it was about working out where you’ve got it wrong’ [D-64].

Confidence Indicator tool –

My final score was fair based on my

I don't care what my score is, I'm moreinterested in Feedback after each

attempt.

I can see how my reflective practicemight be improved by using the

Confidence indicator tool.

My final score was fair based on myconfidence levels.

Confidence Indicator – how useful?

My final score was fair based on my

I don't care what my score is, I'm more

13

0 2 4 6 8 10 12

I can see how my reflective practice

My final score was fair based on my

Number of respondents

Confidence indicator: summary• Pausing to self justify their confidence level encouraged

students to reflect on the evidence for their knowledge• Not all student were good at self assessment and this could • Not all student were good at self assessment and this could

affect their score• The score was valued as a motivator by some, others would

value it more if it could be personalised to reward (applause, stars) rather than penalise

• Such control could encourage more engagement

Confidence indicator: summaryPausing to self justify their confidence level encouraged students to reflect on the evidence for their knowledgeNot all student were good at self assessment and this could

14

Not all student were good at self assessment and this could

The score was valued as a motivator by some, others would value it more if it could be personalised to reward (applause,

Such control could encourage more engagement

Learning logSee my score

15

Read my reflections

Note any learning points, items to revisit, reasons for next

choice of Qs

Log: inserting own reflections

Comments on own errors

Action points

lack of clarity in Qs wording

Qs difficultyReasons for choices

Log: inserting own reflections• Prompted

before leaving each Qs to

16

each Qs to input comments

• The newer the learning, the more they might insert

Feelings

Reasons for choices

Log: review learning pathway

• ‘ I think if you know you’ve got that

Monitor own notes (not score or systems data), • ‘ I think if you know you’ve got that

facility to monitor as you’re going along, that’s useful.’

or systems data), retrace actions,

support follow-up

more likely after incorrect answer (to

‘close the gap’)

fostered feelings of being in control

Log: review learning pathway

I think if you know you’ve got that

17

I think if you know you’ve got that facility to monitor as you’re going along, that’s useful.’ [D]

Personal choiceto use it

Needed familiarisationto adjust to proactive role

Log: review learning pathway• ‘If you’ve got something wrong,

it makes you address it, and look at it again, and that’s how look at it again, and that’s how you learn’ [D]

Other uses: print out, copy and paste details into forums

Log: review learning pathway‘If you’ve got something wrong, it makes you address it, and look at it again, and that’s how

18

look at it again, and that’s how you learn’ [D]

On next visit would like prompt of her previous reflections

I found being able to refer back to the log aneffective tool in helping me w ith my learning.

I found it useful to reflect in the Learning Log w hiledoing the questions.

I found being able to refer back to the log aneffective tool in helping me w ith my learning.

Learning log – how useful?

19

0 2 4 6 8 10 12

I found it useful to reflect in the Learning Log w hile

Number of respondents

Log: summary• Information in one place can aid self• Puts learner in control of providing high quality feedback

themselves themselves – Skills and self-awareness needed– Less experienced learners may not produce high quality

reflections worth revisiting– (Untypically for CBA) responsibility for the quality of the

feedback lies with the learner, rather than the teacher– Such control/responsibility may suit more experienced

learners

Information in one place can aid self-regulation Puts learner in control of providing high quality feedback

20

awareness neededLess experienced learners may not produce high quality

(Untypically for CBA) responsibility for the quality of the feedback lies with the learner, rather than the teacherSuch control/responsibility may suit more experienced

Accessing question• ‘external’ pre-authored

feedback– to support ‘closing the – to support ‘closing the

gap’Correct/incorrect

What the correct answer is & why

Explanation & tips

Reference to relevant course materials

Accessing question-based feedback

21

Feedback: why not referred to?• ‘I thought ‘I’m wrong, what did I do wrong’,

and then I go back myself and try and analyse it myself to find out what went analyse it myself to find out what went wrong’ [C-3]

If answer is correct, additional feedback has no value

• ‘Whatever the situation, if the end result is that I’m going to get a score, and that’s going to make a difference to e.g. my course result, that’s going to be a very important driver for me

Achievementoriented

Feedback: why not referred to?I thought ‘I’m wrong, what did I do wrong’, and then I go back myself and try and analyse it myself to find out what went

Not needed for a confident learner: when clear what mistake is

22

analyse it myself to find out what went mistake is

additional feedback

Whatever the situation, if the end result is that I’m going to get a score, and that’s going to make a difference to e.g. my course result, that’s going to be a very important driver for me’ [C-52]

Feedback: why/when referred to?

• S: ‘apprehensive’ & ‘methodical’ • D: ‘systematic’ & ‘[tending] not to cut corners’• D: ‘systematic’ & ‘[tending] not to cut corners’

Approach dependent on readiness for new/ additional information

• ‘when I got that question wrong I could then read and think that’s why, that was where you were coming from, and it was different to where I was coming from.’

Formative: receptive to

instructional role, transferability

Feedback: why/when referred to?

apprehensive’ & ‘methodical’ not to cut corners’

Less confident learners

Learning-oriented

23

not to cut corners’

Tips should be easily distinguishable from main feedback

hen I got that question wrong I could then read and think that’s why, that was where you were coming from, and it was different to where I was coming from.’ [S-52]

Formative: receptive to

instructional role, transferability

oriented

Feedback: why/when not referred to?• ‘I would look on it as a learning

thing, the fact that I’d actually worked through and got the worked through and got the question right at the end.’ [D-

• ‘…as long as the way I’ve done it gets a correct answer, that’s all I’m interested in.’ [S-4].

Feedback: why/when not referred to?I would look on it as a learning thing, the fact that I’d actually Wanted simply to know

Learning-oriented

24

-3] it was correct (not focusing on score)

Wanted consolidation, not confusing additional methods of doing the same task

Feedback: summary• Learning oriented and confident

> ready to read and act > value feedback to help close gap/ extend learning> value feedback to help close gap/ extend learning– Familiarity with environment needed to encourage use

and high quality self-generated feedback

• Achievement-oriented (score driven) and confident > personal strategies so ignore feedback> not needed to close gap/additional learning is not sought

Feedback: summaryLearning oriented and confident

> value feedback to help close gap/ extend learning

25

> value feedback to help close gap/ extend learningFamiliarity with environment needed to encourage use

generated feedback

oriented (score driven) and confident > personal strategies so ignore feedback> not needed to close gap/additional learning is not

Feedback: implications• After correct answers fuller feedback not necessary? • Score-driven students may circumvent formative

learning opportunities as they focus on summative goals learning opportunities as they focus on summative goals (Miller 2008)

• Learners need skills and awareness to selfhigh quality feedback

• Learners value more personalised control over feedback to help make selective use based on its – Timing– Relevance – Quality

Feedback: implicationsAfter correct answers fuller feedback not necessary?

driven students may circumvent formative learning opportunities as they focus on summative goals

26

learning opportunities as they focus on summative goals

Learners need skills and awareness to self-generate

Learners value more personalised control over feedback to help make selective use based on its

References• Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports

student learning. Learning and Teaching in Higher Education• Miller, Tess (2008). Formative computer

education: the effectiveness of feedback in supporting student learning. education: the effectiveness of feedback in supporting student learning. Assessment & Evaluation in Higher Education,August 22, 2008, from http://www.informaworld.com/10.1080/02602930801956075

• Nicol, D. and Macfarlane-Dick, D. (2006) 'Formative assessment and selfregulated learning: A model and seven principles of good feedback practice', Studies in Higher Education, 31 (2), pp. 199

• The REAP (Re-engineering Assessment Practices)http://www.reap.ac.uk/

• Sadler, R., D. (1998). Formative assessment: revisiting the territory. Assessment in Education, 5(1), 77-84.

Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports Learning and Teaching in Higher Education, 1(1), 3-31.

Miller, Tess (2008). Formative computer-based assessment in higher education: the effectiveness of feedback in supporting student learning.

27

education: the effectiveness of feedback in supporting student learning. Assessment & Evaluation in Higher Education, 34(2), 181-192. Retrieved

http://www.informaworld.com/10.1080/02602930801956075Dick, D. (2006) 'Formative assessment and self-

regulated learning: A model and seven principles of good feedback practice', 31 (2), pp. 199-216.

engineering Assessment Practices) project

Sadler, R., D. (1998). Formative assessment: revisiting the territory.

Forthcoming publication• Nix, I., & Wyllie, A. (In Press) Exploring design features to

enhance computer-based assessment: learners’ views on using a confidence-indicator tool and computerconfidence-indicator tool and computerdoi:10.1111/j.1467-8535.2009.00992.x

LINA demo and project website• http://www.open.ac.uk/colmsct/activities/details/detail.php?itemId=

461cd0ac48c9d

Contact Ali Wyllie - a.j.wyllie@open.ac.uk Ingrid Nix - i.nix@open.ac.uk

Forthcoming publicationNix, I., & Wyllie, A. (In Press) Exploring design features to

based assessment: learners’ views on using a indicator tool and computer-based feedback, BJET,

28

indicator tool and computer-based feedback, BJET, 8535.2009.00992.x

http://www.open.ac.uk/colmsct/activities/details/detail.php?itemId=

a.j.wyllie@open.ac.uk i.nix@open.ac.uk