Assessing with confidenceJon Rosewell

Confidence-based question

Why ask students about certainty?

Ask students:

●What do you know?

●How certain is your knowledge?

Wikipedia: https://en.wikipedia.org/wiki/Confidence-based_learning

How can we get students to honestly report their certainty?

Confidence ≡ certainty

Confidence-based marking (CBM)

NB terminology: ‘certainty’ would be better, but ‘confidence’ has stuck

ConfidenceScore

Correct Wrong

Low 1 0

Medium 2 -2

High 3 -6

Tentative & correct

Confidently correct

Cocksure – and wrong!

Gardner-Medwin & Curtin (2007) Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment[http://www.ucl.ac.uk/lapt/REAP_cbm.pdf]

Some trial questions

Question Taken by Mean scoreWhat is 2 + 2? 238 2.51What is derivative of x³? 223 -0.47Who painted the 'Mona Lisa'? 212 2.14Who is the 'Mona Lisa'? 208 0.21Uncertainty principle -- whose? 207 0.03Uncertainty principle -- formula? 218 0.01

Easy

Difficult

Tricky!

Do students honestly assess confidence?

Question High Medium LowWhat is 2 + 2? 210 5 23What is derivative of x³? 96 46 81Who painted the 'Mona Lisa'? 165 21 26Who is the 'Mona Lisa'? 56 41 111Uncertainty principle -- whose? 62 31 114Uncertainty principle -- formula? 39 21 158

Medical students

CBM – for learning and revision

Moodle: https://docs.moodle.org/30/en/Using_certainty-based_marking

‘CB bonus’

Under exam conditions…

Benefits to students

CBM – motivations

●Rewards care and effort

●Greater engagement

●Encourages reflective learning

●Encourages self-assessment Thinkstock.com: 483452995

Do students like CBM?

Yes – regard it as fair and challenging, helpful to learning

and

No – less likely to do CBM than MCQ when optional

Schoendorfer, N., & Emmett, D. (2012). Use of certainty-based marking in a second-year medical student cohort: a pilot study. Advances in Medical Education and Practice, 3, 139–43. doi:10.2147/AMEP.S35972Nix, I., & Wyllie, A. (2011). Exploring design features to enhance computer-based assessment: Learners’ views on using a confidence-indicator tool and computer-based feedback. British Journal of Educational Technology, 42(1), 101–112. doi:10.1111/j.1467-8535.2009.00992.xBarr, D. A., & Burke, J. R. (2013). Using confidence-based marking in a laboratory setting: A tool for student self-assessment and learning. The Journal of Chiropractic Education, 27(1), 21–26. doi:10.7899/JCE-12-018

Is CBM fair?

●No significant gender differences

●Very few students seem over-confident, but some were under-confident‘In decision-rich occupations such as medicine, mis-calibration of reliability is a

serious handicap’ Gardner-Medwin (2014, p.6)

●Scores will generally be lower when marked as CBM than MCQbut possible to scale to non-CBM marking to set grade boundaries

Gardner-Medwin, A. R., & Gahan, M. (2003). Formative and Summative Confidence-Based Assessment. In Proc. 7th International Computer-Aided Assessment Conference (pp. 147–155). Retrieved from www.caaconference.com Gardner-Medwin, T. (2014). CBM selftests at UCL: The past and the future of LAPT. Retrieved from http://www.tmedwin.net/~ucgbarg/tea/SLMS2014_A4.pdf

Types of questions

Open question Multiple-choice question

Confidence-based

What is wrong with MCQ?

MCQPros: Objective marking Reliable marking Easy to implement

Cons: Distractors trivial May engender

misconceptions Working backwards

Open questionPros: Numeric easy to mark Tests deeper learning Can find misconceptions

Cons: Free text still difficult to

mark reliably

‘Open’ CBM

‘Open’ CBM – what benefit?

MCQPros: Objective marking Reliable marking Easy to implement

Cons: Distractors trivial May engender

misconceptions Working backwards

Open CBMPros: Open question Reflection …as MCQ

Cons: Not always applicable Intimidating? Personality dependent?

17

www.indegene.com/lifesciences/ipace

ipace

●‘Global training partner to pharma companies’

●Need to establish that reps are properly trained for compliance

●Platform that delivers:●Regular questions from a bank

●CBM assessment

●Mastery = all questions correctly answered more than once

18

19

20

21

22

23

24

Future of CBM?

Pros:

●Fit with competency and mastery assessment is a good one

●Use in formative / revision contexts avoids issues to do with

unconventional marking and assigning grade boundaries

Cons:

●Dislike of negative marking

●Poor platform support – but improving

●Difficulty of marking more complex question types

26