The Continual Assessment of The Continual Assessment of Confidence or Knowledge with Confidence or Knowledge with
Hidden MCQ?Hidden MCQ?
Short Paper W.I.P.Short Paper W.I.P.
Phil DaviesPhil Davies
School of ComputingSchool of Computing
University of GlamorganUniversity of Glamorgan
Conference 1999Conference 1999Learning Through AssessmentLearning Through Assessment
Phil DaviesPhil Davies
School of ComputingSchool of Computing
University of GlamorganUniversity of Glamorgan
Implementation of OLALImplementation of OLAL
Tests Weeks 3,6,8,12 (60% Subject Mark)Tests Weeks 3,6,8,12 (60% Subject Mark) Percentage Marks 5%,15%,15%,25%Percentage Marks 5%,15%,15%,25% ““Beep Test”Beep Test” +2 Correct -1 Incorrect(Really -3)+2 Correct -1 Incorrect(Really -3)
Student ConcentrationStudent Concentration
GuessingGuessing– Why should the student care if incorrect?Why should the student care if incorrect?
Allow TWO PASSES of the testAllow TWO PASSES of the test By concentrating on first passBy concentrating on first pass
– will get reward on second passwill get reward on second pass– formativeformative
Results of The TestsResults of The Tests
Improvement of Test Results Pass One to Two
Who benefited from the test?Who benefited from the test?
<10 10-19
20-29
30-39
40-49
50-59
60-69
70-79
80-89
90>
0 1 5 5 20 23 27 17 10 0
Student FeedbackStudent Feedback
Excellent way of gaining knowledgeExcellent way of gaining knowledge 2nd Test Allowed us to learn2nd Test Allowed us to learn Good assistant to learningGood assistant to learning Encourages RevisionEncourages Revision Very good, promotes learning of a subject Very good, promotes learning of a subject
early onearly on Better than sex!!!!Better than sex!!!!
““There’s no confidence in There’s no confidence in multiple-choice testing multiple-choice testing
…”…”
CAA Conference 2002CAA Conference 2002
Phil DaviesPhil Davies
What decides the QUALITY of a What decides the QUALITY of a question?question?
In the 2001 World cup qualifier between In the 2001 World cup qualifier between England and Greece, who scored the all England and Greece, who scored the all important equalizing goal?important equalizing goal?
David BeckhamDavid Beckham Victoria BeckhamVictoria Beckham Brooklyn BeckhamBrooklyn Beckham Rebecca LoosRebecca Loos
Rewards?Rewards?
Know itKnow it On the tip of my tongueOn the tip of my tongue If I see the answers I’ll know itIf I see the answers I’ll know it DeductionDeduction GuessGuess Know it … oh no I’ve got it wrong .. Know it … oh no I’ve got it wrong ..
misinformationmisinformation
Proportion of Questions CorrectProportion of Questions Correct
High Confidence => 84% correctHigh Confidence => 84% correct Fairly Confident => 58% correctFairly Confident => 58% correct Not confident => 47% correctNot confident => 47% correct
72% of all questions answered with high 72% of all questions answered with high confidenceconfidence
Student thoughts on ConfidenceStudent thoughts on Confidence
““Eliminates guesswork”Eliminates guesswork” ““I was waiting for a leggy blond to bring I was waiting for a leggy blond to bring
on my prize at the end”on my prize at the end” ““separated the lucky students from me”separated the lucky students from me” ““at least I didn’t lose marks for selecting at least I didn’t lose marks for selecting
no confidence”no confidence” ““are you testing my confidence or are you testing my confidence or
knowledge”knowledge”
Quality of a multiple choice Quality of a multiple choice question?question?
In the 2001 World cup qualifier between In the 2001 World cup qualifier between England and Greece, who scored the all England and Greece, who scored the all important equalizing goal?important equalizing goal?
Teddy SheringhamTeddy Sheringham Michael OwenMichael Owen Robbie FowlerRobbie Fowler QUALITY OF QUESTION NOT QUALITY OF QUESTION NOT
DISTRACTERSDISTRACTERS
““Bite The Bullet”Bite The Bullet”
Use Confidence Testing for all four tests (10%, Use Confidence Testing for all four tests (10%, 15%, 20% & 25%)15%, 20% & 25%)
Weighting of Passes (50/50, 60/40, 70/30 & 80/20)Weighting of Passes (50/50, 60/40, 70/30 & 80/20) How to judge if the students had ‘really’ learned?How to judge if the students had ‘really’ learned? Identify improvement > Identify improvement > Select five questions that were answered ‘poorest’ Select five questions that were answered ‘poorest’
in test one & pass through to test twoin test one & pass through to test two Select five questions that were answered ‘poorest’ Select five questions that were answered ‘poorest’
in test two & pass through to test three ( + original in test two & pass through to test three ( + original five) …. Etc.five) …. Etc.
How to identify ‘Poorest’How to identify ‘Poorest’
Using confidence, degree of correctness of Using confidence, degree of correctness of answeringanswering
Using confidence scores to provide a Using confidence scores to provide a degree of ‘poorness’ degree of ‘poorness’ i.e. +4, +2, +1, -6, -5, -4i.e. +4, +2, +1, -6, -5, -4
High degree of ‘poorness’ could just be a High degree of ‘poorness’ could just be a ‘BAD’ question‘BAD’ question
Swing between 1Swing between 1stst and 2 and 2ndnd pass of degree of pass of degree of correctness should be ‘Reasonable’correctness should be ‘Reasonable’
80
>
75
-79
70
-74
65
-69
60
-64
55
-59
50
-54
45
-49
40
-44
35
-39
30
-34
25
-29
1 2 5 6 11 13 15 12 7 7 6 3
Test # Average Pass One Average Pass Two Ratio First/Second Average Total
Test One10%
42.56% 72.74% 50 / 50 57.65%
Test Two15%
26.80% 66.62% 60 / 40 42.73%
Test Three20%
41.14% 76.40% 70 / 30 51.72%
Test Four25%
51.00% 81.50% 80 / 20 57.10%
Test 2 - 1 Test 3 - 2 Test 4 - 3 Test 4 - 1
80> -2 4 23 25.00
75-79 10.5 9 45.5 65.00
70-74 -31.8 43 9.6 20.80
65-69 -34.83 43.83 29.67 38.67
60-64 -45.27 53.27 16.55 24.55
55-59 -43.15 36.08 20.23 13.15
50-54 -36.2 48.8 21.53 34.13
45-49 -7.0 27.92 38.83 59.75
40-44 -45.43 30.00 28.43 13.00
35-39 -32.86 27.00 25.71 19.86
30-35 -47.33 44.5 31.33 28.50
25-29 -47.67 37.00 -13.00 -23.67
Average -30.25 +33.70 +23.12 +26.56
Identifying Poor Questions Test Identifying Poor Questions Test OneOne
Qu
estio
n N
um
be
r
# C
orre
ct Pa
ss 1
# W
ron
g P
ass 1
We
igh
ting
Fa
ctor
# C
orre
ct Pa
ss 2
# W
ron
g P
ass 2
We
igh
ting
Fa
ctor
We
igh
ting
Diffe
ren
ceP
ass 2
-1
Q5 27 61 -251 56 32 +9 260
Q14 30 58 -218 60 28 +67 285
Q15 34 54 -181 55 33 +4 185
Q19 8 80 -356 76 12 +223 579
Q20 33 55 -187 58 30 +16 203
Q27 11 77 -373 58 30 +21 394
Identifying the Poorest Identifying the Poorest Questions Test two to threeQuestions Test two to three
Qu
estio
n N
um
be
r
# C
orre
ct Pa
ss 1
# W
ron
g P
ass 1
We
igh
ting
Fa
ctor
# C
orre
ct Pa
ss 2
# W
ron
g P
ass 2
We
igh
ting
Fa
ctor
We
igh
ting
Diffe
ren
ceP
ass 2
-1
Q2 14 74 -348 79 9 250 598
Q6 28 60 -222 45 43 -109 331
Q8 22 66 -255 70 18 145 400
Q10 28 60 -215 48 40 -66 281
Q11 20 68 -281 57 31 20 301
Q18 28 60 -200 79 9 227 427
Q19 31 57 -202 74 14 166 368
Q21 27 61 -235 57 31 32 267
Q27 32 56 -210 66 22 106 316
Q28 27 61 -256 77 11 238 494
Q32 23 65 -254 82 6 265 519
Identifying Poorest Questions Identifying Poorest Questions from Three to Fourfrom Three to Four
Qu
estio
n N
um
be
r
# C
orre
ct Pa
ss 1
# W
ron
g P
ass 1
We
igh
ting
Fa
ctor
# C
orre
ct Pa
ss 2
# W
ron
g P
ass 2
We
igh
ting
Fa
ctor
We
igh
ting
Diffe
ren
ceP
ass 2
-1
Q6 32 56 -212 63 25 73 285
Q11 27 61 -219 75 13 217 436
Q16 19 69 -313 60 28 58 371
Q26 30 58 -225 81 7 253 480
Q40 30 58 -218 72 16 183 401
Passed through to Test FourPassed through to Test Four
13 of the poorest question13 of the poorest question
6 new questions based upon additional work 6 new questions based upon additional work between tests 3-4between tests 3-4
21 questions (cross section of questions 21 questions (cross section of questions and topics from previous tests)and topics from previous tests)
Improvement of Results Through Improvement of Results Through TestsTests
Questions Through Tests
-500
-400
-300
-200
-100
0
100
200
300
400
1 2 3 4 5 6 7 8
Question 5
Question 6
Question 10
Question 11
Question 14
Question 15
Question 16
Question 19
Question 20
Question 21
Question 26
Question 27
Question 40
Improvement Pass One of QuestionsImprovement Pass One of Questions
Pass One Question Improvements
-500
-400
-300
-200
-100
0
100
200
300
400
1 2 3 4
Question 5
Question 6
Question 10
Question 11
Question 14
Question 15
Question 16
Question 19
Question 20
Question 21
Question 26
Question 27
Question 40
Improvement for Question 5Improvement for Question 5
Stronger Students improved quickly through Stronger Students improved quickly through the teststhe tests
Became more confident in answering the Became more confident in answering the questionsquestions
Weaker Students improved less quickly than Weaker Students improved less quickly than stronger through the test progressstronger through the test progress
Is it just Confidence or Knowledge Is it just Confidence or Knowledge i.e. questions right or wrong?i.e. questions right or wrong?
Category Average # of Questions Correct Pass 1
Average # of Questions Correct Pass 2
Average Total # of Questions Correct
80> (1) 131.0 158.0 289.0
75-79 (2) 123.5 150.5 274.0
70-74 (3) 117.6 148.6 266.2
65-69 (4) 113.8 150.7 264.5
60-64 (5) 110.3 145.0 255.3
55-59 (6) 107.6 143.4 251.0
50-54 (7) 100.4 136.2 236.6
45-49 (8) 92.5 132.6 225.1
40-44 (9) 94.0 126.4 220.4
35-39 (10) 87.7 121.6 209.3
30-34 (11) 80.5 117.3 197.8
25-29 (12) 76.7 117.3 194.0
Some ConcernsSome ConcernsCategory Average # of Questions
Correct Pass 1Average # of Questions
Correct Pass 2Average Total # of
Questions Correct
12 81 127 208 (194)
3 102 136 238 (266)
7 78 117 195 (236)
8 112 141 253 (225)
10 97 137 234 (209)
Overall CommentsOverall Comments
Work in ProgressWork in Progress Results look very positiveResults look very positive Correctness Factor is dependant upon weighting Correctness Factor is dependant upon weighting
of scoresof scores Guessing not so much of a problem Guessing not so much of a problem Changed students mode of revising / studyChanged students mode of revising / study Fine granularity in the QUALITY OF A QUESTIONFine granularity in the QUALITY OF A QUESTION
Possibility of Computerized Adaptive TestingPossibility of Computerized Adaptive TestingHowever note the quality of the question change However note the quality of the question change
throughout the course of the cumulative teststhroughout the course of the cumulative tests Does gender come into it??Does gender come into it??
Contact InformationContact Information
Email: Email: [email protected]@glam.ac.uk Phone: 01443 482247Phone: 01443 482247
School of ComputingSchool of ComputingUniversity of GlamorganUniversity of GlamorganPontypriddPontypriddMid – GlamorganMid – GlamorganSouth WalesSouth Wales