timing is everything - using psychology research to make your assessments more effective

Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2012 Users Conference New Orleans March 20 - 23

Timing is Everything Using psychology research to make your

assessments more effective John Kleeman

Questionmark Chairman Twitter: @johnkleeman

http://twitter.com/

2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark

is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 2

Would you like to reduce learner forgetting curves?


is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Introduction

Spacing of learning

Using spacing to make assessment more effective

Retrieval practice

Using retrieval practice to make assessment more effective

Other interesting research – self-explanations, feedback

Where to find out more

Questions

Agenda

Slide 3

Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2012 Users Conference New Orleans

Typical Psychology Method

Group A Group B

Slide 4

Learn in another way

Wait a period

Measure learning

Learn in one way

Wait a period

Measure learning

Is learning improved?



Spacing of Learning

Slide 5



Spacing learning aids retention Sp

acin

g M

asse

d



25 students learning difficult English words with flashcards Brilliant -> Effulgent Righteousness -> Rectitude

Massed Learn ¼ words each day for 4 days. Review before test

Spaced Spend same time Look at all words each day for 4 days Review before test

One experiment : Kornell 2009

Slide 7



Spacing more

effective than massed / cramming

Most participants predicted the opposite

Results

Slide 8



Shebilske + colleagues, 1999

Used video game useful for pilot training

20 people learn massed 10 lessons 2 days

20 people learn spaced 10 lessons 10 days

Control for differences

A classic experiment

Slide 9



Massed group Mean score 2,156 after

10 lessons Distributed group Mean score 3,189 after

10 lessons 48% better

Participants were tested a week later and the effect remained

Experiment results

Slide 10



Spacing Repeat learning material, separated over time Effective for long term retention

Cramming Repeating learning intensely (e.g. before exam) Effective to pass an exam, not for long term

Interleaving topics More effective than massing them

Spacing effect

Slide 11



Using assessments to take advantage of the spacing effect



Quizzes + feedback = learning Can be spaced to give repetition Some ideas: Quiz of the week Question of the day Quiz with evaluation survey Quizzes as post-learning reinforcement Quizzes on mobile devices

Use assessments as spaced learning



One exam at end of course Encourages cramming Does not measure long term retention

Tests or quizzes during course Potentially promote revision & learning at spaced intervals Which increase retention Potential techniques Mock/practice exams Regular quizzes with small amounts of credit Splitting up exam into series over time

Use assessments to promote spaced learning

Slide 14



Evidence? One study from Angus & Watson

Slide 15



Your questions on spacing

Slide 16



Retrieval Practice

Slide 17



Francis Bacon, 1620

If you read anything over twenty times, you will not learn it by heart so easily as if you were to read it only ten, trying to repeat it between whiles, and when memory failed, looking at the book.

Retrieval helps you learn

Slide 18



Learning

Slide 19

Receive information

Working memory

Long term memory



Cue • Question • Other stimulus

Search • Retrieve from long

term memory

Act • Act or respond

Research shows retrieval practice helps future retrieval

Successful retrieval modifies memory making it more retrievable in future

Learning useful if can retrieve

Slide 20



120 students divided into groups randomly Given reading comprehension passages used in TOEFL Some students Study for 14 mins

Some students Study for 7 mins Recall material for 7 mins

Compared results after 5 mins and 1 week

Roediger & Karpicke experiment 2006

Slide 21



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

After 5 mins After 1 week

RestudyQuiz

How much learners retained

Slide 22



0%

10%

20%

30%

40%

50%

60%

SAT II Bloom Definition Bloom Application

No quizQuiz

Marsh & others 2007. Experiments with multiple choice questions

Slide 23



80 students in 4 groups, studying a science text Study Repeated Study Concept Mapping (mind maps) Retrieval Practice

Surveyed learners on expectation of retention Tested a week later to see how much retained

Karpicke & Blunt experiment 2011

Slide 24



0%10%20%30%40%50%60%70%80%

Actual retention

0%10%20%30%40%50%60%70%80%

Learner expectation of retention

How much learners retained

Slide 25



Larson experiment 2009 Quizzes reduce forgetting curve

In real-life, not in the laboratory Medical residents (doctors in training) 20 doctors in each group

Quiz group Restudy group

Day 1 Study Study

Day 1 Quiz Re-study

2 weeks later Quiz Re-study

2 weeks later Quiz Re-study

6 months later Final exam Final exam



62%

44% 48%

36%

62%

19%

Initial Quiz Quiz 6 months

Quiz group Re-study group

Part of the results

Slide 27



Applying retrieval practice results for assessments

Slide 28



Retrieval practice helps retention of learning Applies to facts And also concepts

Retrieval practice includes Recall Formative quizzes/tests Self-assessment Explain or read back of material

Timing of retrieval Works best when spaced out Strongest effect when retrieval is difficult

Feedback very helpful Repetition of retrieval Research suggests 5-7 spaced repetitions helpful

Retrieval practice key findings

Slide 29



Learners do not realize they will forget Quizzes encourage/force retrieval practice

Quizzes give corrective feedback And also: Randomize so easy to repeat Mobile devices Accessible Measure that practice has happened

Computer quizzes are effective retrieval practice because …

Slide 30



Give quizzes after learning sessions Focus questions on important things to remember Make quizzes effortful (delay & difficulty) Question types: Recall (e.g. short answer) best Multiple choice work too

Include feedback, usually the correct answer Encourage retrieval (don’t allow peek ahead) Spacing helps significantly

If you are a practitioner …

Slide 31



How easy is it for learners to take quizzes? How easy is it for your instructors to quiz learners? Could you make it easier?

If you are a manager …

Slide 32



Your questions on retrieval

Slide 33



Self-explanations

Slide 34



Learners who can self-explain something learn it better Inferences Explain why something is What the principle behind an example is

Atkinson + others did an experiment in mathematics

learning proving self-explanation questions can help learning

Self-explanations



Conventional, non self-explanation question

Slide 36



Self-explanation question that helps learning better

Slide 37



Their conclusions More effective than other kinds of questions Works best in disciplines like maths and science where there

are clear principles Gives good scaffolding to help people learn why and not what

Possible application Consider questions which give the answer and ask

participants to identify the reason why it is so

Self-explanations

Slide 38



Psychological Research on Feedback

Slide 39



Where does feedback fit in?

Slide 40

Current State

Search / Retrieval

Response Evaluation

Adjust



0%

10%

20%

30%

40%

50%

60%

No quiz Quiz, no feedback Quiz, immediatefeedback

Quiz, delayedfeedback

Retention after one week

Roediger & Butler. Feedback on multiple choice questions

Slide 41



Students only know wrong if you tell them Include correct answer In most contexts, just telling people they are wrong doesn’t

help learning Feedback most useful on incorrect answers Some value on correct answers, especially if learner guesses

Feedback tip #1 – give feedback in formative assessments

Slide 42



Usually force people to retrieve before you give feedback Don’t allow them to peek ahead to see the right answer This deprives them retrieval practice

Exception when building understanding

Feedback tip #2 – do retrieval first

Slide 43



Feedback only useful if learners pay attention

Give feedback so that people will attend Consider attention span Vary feedback methods (e.g. graphics) Review whether to give feedback on correct answers Monitor effect of feedback

Keep feedback simple, clear and not too long

Feedback tip #3 – get people to pay attention

Slide 44



Slight delay on feedback is usually better than immediate feedback for retention Probably due to spacing effect in learning End of test / after each block feedback better than after each

question feedback Research not conclusive

Feedback tip #4 – delay feedback if you can

Slide 45



In a Perfect World

Slide 46


is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 47

Learning in the Actual World They don’t get it All!



Learning Curve

Slide 48

Ouch! The Forgetting Curve

Time

Know

ledg

e/Sk

ills

Lear

ning

Ex

perie

nce

Lea

rnin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Que

stio

ns

Que

stio

ns

Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall

Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall



In the Actual World They Forget It!

Slide 49



Slow the Forgetting Curve

Slide 50

Time

Know

ledg

e/Sk

ills

Lear

ning

Ex

perie

nce

Lea

rnin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Que

stio

ns

Que

stio

ns

Que

stio

ns

Que

stio

ns

Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve

Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve



Questions through Learning Process

Slide 51

Time

Know

ledg

e/Sk

ills

Que

stio

ns

Que

stio

ns

Que

stio

ns

Que

stio

ns

Q

uest

ions

Create Intrigue

Strengthen Memory

Reduce Forgetting



Something that Slows initial learning Increases long-term retention and transfer

Makes you think you are learning less well, but in fact allows you to retain information

Examples Testing rather than re-studying Spaced learning Interleaving tasks Delayed feedback

Paradox : to learn better, we may need to think we are learning less well

Paradox : The best time to reinforce learning is when we are just about to forget something

“Desirable Difficulty”

Slide 52



Where to find out more

Slide 53



Building Expertise Ruth Colvin Clark

Applying the Science of Learning Richard Mayer

Two recommended books

http://www.amazon.com/gp/product/images/0136117570/ref=dp_image_0?ie=UTF8&n=283155&s=books



Expert at translating research into practice “Learning takes place over time—real learning doesn’t usually occur

in one-time events”

Two Questionmark white papers www.questionmark.com/whitepapers Learning Benefits of Questions Providing Feedback to Learners

Good material on own site

www.work-learning.com Spacing Learning over Time

Dr Will Thalheimer

Slide 55

http://www.questionmark.com/whitepapers

http://www.work-learning.com/



Who Title Journal Date Description URL Kornell Optimising Learning Using

Flashcards: Spacing Is More Effective Than Cramming

Appl. Cognit. Psychol 2009 Spacing expemiment Here

Shebilske, Goettl, Corrington, Day

Interlesson Spacing and Task-Related Processing During Complex Skill Acquisition

Journal of Experimental Psychology: Applied

1999 Space Fortress experiment Has to be purchased

Angus & Watson

Does regular online testing enhance student learning in the numerical sciences? Robust evidence from a large data set

British Journal of Educational Technology Vol 40 No 2, 255-272

2009 Correlation study See here for info

Thalheimer Spacing Learning over time 2006 Good general explanation www.work-learning.com

References – spacing

http://web.williams.edu/Psychology/Faculty/Kornell/Publications/Kornell.2009b.pdf

http://blog.questionmark.com/wheres-the-evidence-for-assessment

http://www.work-learning.com/catalog.html

http://www.work-learning.com/catalog.html



Who Title Journal Date Description URL Roegiger & Karpicke

Test-Enhanced Learning Psychological Science 2006 Widely cited paper showing benefits of testing vs studying

Here

Marsh, Roediger, Bjork&Bjork

The memorial consequences of multiple-choice testing

Psychonomic Bulletin & Review

2007 Shows benefits of multiple choice questions

Here

Larsen, Butler & Roediger

Test-enhanced learning in medical education.

Medical Education 2009 Shows experiment on medical education and retrieval practice

Here

Roediger & Butler

The critical role of retrieval practice in long-term retention

Trends in Cognitive Sciences

2010 8 page review article on retrieval and retention

Here

Karpicke & Blunt

Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping

Science Magazine 2011 Clear paper comparing retrieval with study and mind maps

Select from here (non-commercial)

Kleeman Recent cognitive psychology research shows strongly that quizzes help retain learning. What does this mean for CAA?

International CAA Conference

2011 Some more detail from me Here

References – retrieval practice

http://psych.wustl.edu/memory/Roddy article PDF's/Roediger&Karpicke(2006)Psych_Sci.pdf

http://psych.wustl.edu/memory/Roddy article PDF's/MarshRoedigerBjork&Bjork2007.pdf

http://psych.wustl.edu/memory/Roddy article PDF's/Larsen,Butler&Roediger2008.pdf

http://duke.edu/~ab259/pubs/Roediger&Butler(2010).pdf

http://memory.psych.purdue.edu/publications/

http://caaconference.co.uk/wp-content/uploads/Kleeman-CAA2011.pdf



Who Title Journal Date Description URL

Atkinson, Renkl & Merrill

Transitioning From Studying Examples to Solving Problems

Journal of Educational Psychology

2003 Paper on self-explanations

Here

Valerie J. Shute Focus on Formative Feedback ETS (Educational Testing Service)

2007 45 page article giving advice on feedback

ETS website

Thalheimer Providing Learners with Feedback

Questionmark white paper

2008 Two part white paper on effective feedback

Questionmark website: Part 1 Part 2

Fazio, Huelser, Johnson & Marsh

Right/wrong feedback: Consequences for learning

Memory 2010 Research showing just telling students right or wrong without correct answer unhelpful

Here

References – self-explanations & feedback

Slide 58

http://www.cs.pitt.edu/~chopin/references/tig/AtkinsonRenklM_03.pdf

http://www.ets.org/Media/Research/pdf/RR-07-11.pdf

http://www.questionmark.com/catalog/us/resources/Providing_Learners_with_Feedback_Part1.pdf

http://www.questionmark.com/catalog/us/resources/Providing_Learners_with_Feedback_Part2.pdf

http://pn.aas.duke.edu/sites/marsh/Fazio_Huelser_Johnson_Marsh_2010.pdf

2012 Users Conference New Orleans

Your questions

You can also email me at [email protected] or follow me on Twitter @johnkleeman

Slide 59

mailto:[email protected]

http://twitter.com/

timing is everything - using psychology research to make your assessments more effective

Technology

time effective

exam effective

effective retrieval

repeating learning

days review

advantage ofthe spacing

massed groupmean score

days control