The Moodle Quiz at the Open University: how we use it & how that helps students

Tim Hunt, Information Technology

MoodleMoot Dublin 2015

The Moodle Quiz at the Open University: how we use it & how that helps students

Tim Hunt, Information Technology

MoodleMoot Dublin 2015many dedicatedOU teaching staff

Alternate titles

Lots of interesting graphs

The scholarship of teaching and learning in practice

Overall use

#PartTimeMatte

rs

It matters

End of module survey results

E100 E101 E105 E111 E112 E207 E210 U2110%

20%

40%

60%

80%

100%

KPI 01: Overall, I am satisfied with the quality of this module.Q33 Clear understanding of what was required to complete the assessed workQ34 the assessment activities supported my learning

It makes a difference

Warning!

Quantum

mechanics

ahead

Level 3 physics (SM358)

SM358 assessment tasks

0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs

0 iCMAs

1 iCMA

2 iCMAs

3 iCMAs

4 iCMAs

5 iCMAs

6 iCMAs

Students completing tasks

0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs

0 iCMAs 11.6% 3.4% 1.5% 0.5% 0.5%

1 iCMA 1.5% 1.0%

2 iCMAs 1.5% 2.4% 1.5%

3 iCMAs 1.5%

4 iCMAs 5.3% 2.4%

5 iCMAs 0.5% 3.9% 5.8% 8.2%

6 iCMAs 0.5% 0.5% 5.8% 5.8% 34.3%

Average exam scores

0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs

0 iCMAs 6.0

1 iCMA

2 iCMAs 17.0 24.0

3 iCMAs 60.0

4 iCMAs 43.7 62.0

5 iCMAs 23.0 46.0 62.6 69.5

6 iCMAs 35.3 60.8 77.5

Exam scores vs prediction

0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs

0 iCMAs −20.8

1 iCMA

2 iCMAs −43.9 −27.5

3 iCMAs −9.0

4 iCMAs −15.6 +1.8

5 iCMAs −3.8 −11.1 +1.4 +2.4

6 iCMAs −17.1 +3.4 +4.6

Changing assessment type

T184 robotics & meaning of life

Before

10% mid-course iCMA

90% final written EMA

part 1 short-answer

part 2 programming

& essays

After

10% mid-course iCMA

30% final iCMA

60% final written EMA

Module completion ratesT184 completion rates

60%

65%

70%

75%

80%

85%

90%

95%

100%

2004 2005 2006 2007 2008 2009 2010 2011

May

Oct

Introduction of CME

T184 completion rates

60%

65%

70%

75%

80%

85%

90%

95%

100%

2004 2005 2006 2007 2008 2009 2010 2011

May

Oct

Introduction of CME

Module completion rates

T184 completion rates

60%

65%

70%

75%

80%

85%

90%

95%

100%

2004 2005 2006 2007 2008 2009 2010 2011

May

Oct

Introduction of CME

Module completion rates

Deadlines

SM358 iCMA51 submit date

2010 advisory deadline

SM358 iCMA51 submit date

2010 advisory deadline2012 hard deadline

Grades

Optional quizzes

Compulsory quizzes

Do students read feedback?

Interactive with multiple tries

Same question in two quizzes

Can computers grade sentences?

Spoiler: Yes!

How good are humans?

Question

Number of responses in

analysis

Percentage of responses where the human markers

were in agreement with question author

Percentage of responses where

computer marking was in agreement with question author

Range for the 6 human

markers

Mean of the 6 human markers

A 189 97.4 – 100. 98.9 99.5

B 248 83.9 – 97.2 91.9 97.6

C 150 80.7 – 94.0 86.9 94.7

D 129 91.5 – 98.4 96.7 97.6

E 92 92.4 – 97.8 95.1 98.9

F 129 86.0 – 97.7 90.8 97.7

G 132 66.7 – 90.2 83.2 89.4

Comparing three algorithms

Question

Number of responses in

analysis

Percentage of responses where computer marking was in agreement with question

author

Computational linguistics

Algorithmic manipulation of keywords

IAT Pattern-match

Regular Expressions

A 189 99.5 99.5 98.9

B 248 97.6 98.8 98.0

C 150 94.7 94.7 90.7

D 129 97.6 96.1 97.7

E 92 98.9 96.7 96.7

F 129 97.7 88.4 89.2

G 132 89.4 87.9 88.6

Summary

References• Overall iCMA usage numbers collated by Phil Butcher.• End of module server results from

Student Analytics in the Institute of Educational Technology via Linda Price.• SM358 data from John Bolton• T184 data from Jon Rosewell• SDK125, S141, S151 & S240 data from Sally Jordan• Jordan, Sally (2014). Using e-assessment to learn about students and learning.

International Journal of e-Assessment, 4(1) • Jordan, Sally (2014).

Adult science learners’ mathematical mistakes: an analysis of responses to computer-marked questions. European Journal of Science and Mathematics Education, 2(2) pp. 63–86.

• Jordan, Sally (2012). Short-answer e-assessment questions : five years on. In: 2012 International Computer Assisted Assessment Conference, 10-11 July 2012, Southampton.

• Pattern-match question type for Moodle https://moodle.org/plugins/view/qtype_pmatch

Key points

Getting the assessment right is important

Online quizzes can be powerful learning tools

Dig into the data and you can learn• Is this quiz working?• What are my students’ misconceptions?

Computers can grade much more than multi-choice but only on behalf of a teacher

Want more? I’m giving this talk again at #iMoot15