Using interactive computer-based assessment to support beginning distance learners of science

Sally Jordan27th October 2010

Centre for Distance Education Research in Distance Education

My plan

• What we did

• Why we did it

• How we did it

• Evaluation

• How far is it appropriate to go?

What we did 1

• Interactive computer-marked assignments (iCMAs) as part of the assessment strategy of a module, alongside tutor-marked assignments;

• iCMAs are summative but lightly weighted;• Their primary purpose is formative;• Other modules use iCMAs in a purely formative way,

with students being able to repeat questions and try different variants of them for extra practice.

• S104 website iCMA42

Why we did it 1We use interactive computer marked assignments (iCMAs) alongside tutor-marked assignments (TMAs) to:

• Provide instantaneous feedback – and an opportunity for students to act on that feedback;

• Provide ‘little and often’ assessment opportunities and so help students to pace their studies;

• Act as ‘a tutor at the student’s elbow’;• To help students, in conjunction with their tutor, to

monitor their own progress.

We use a range of question types, going beyond those where students select from pre-determined answers to those where they have to write in their own words.

What we did 2

• Diagnostic materials to help students find the correct starting point in their Open University study;

• In the Science Faculty we initially used printed ‘Are you ready for?’ materials;

• We are moving towards interactive diagnostic materials, e.g. ‘Are you ready for science study?’/’Are you ready for S104?’, together used by more than 3000 people every month.

Study and the OU website AYRF S104

Why we did it 2• The Open University operates an open entry policy to its

undergraduate modules;• Getting students on the correct module in the first place

is the most important factor in their subsequent retention, progression and success;

• We want people to make decisions on the basis of their prior knowledge and skills, but these can be difficult to quantify;

• The quizzes advise on the basis of what people can actually do rather than what they think they can do;

• Making the quizzes online and interactive means that students have to engage with the questions rather than just looking at the answers.

How we did it

• Most of our questions are written in OpenMark and sit within the Moodle virtual learning environment;

• For short-answer free-text questions, we initially used answer-matching software provided by Intelligent Assessment Technologies (IAT), now we use PMatch (part of OpenMark);

• Real student responses were used in developing the answer matching.

• Questions were improved in the light of comments from students and tutors and in the light of data-analysis.

Evaluation

Has included:• Human-computer marking comparison• Computer-computer marking comparisons;• Questionnaires and interviews;• Student observation (in usability laboratory);• Data-analysis : used to explore factors that impact on

student engagement with e-assessment;• Further questionnaires and interviews.

Some conclusions• Students seem to like iCMAs: 90% of students agree with the statement ‘Answering

iCMA questions helps me learn’.• Examples of comments from students:

‘iCMAs are a brilliant tool in building confidence’

‘it’s more like having an online tutorial than taking a test’.• And from a tutor

‘a great example of how online assessment can aid learning.’

• But minor errors and ambiguities in individual questions can be off-putting to students, so it is important to monitor use.

Some conclusions cont.• Students appear to engage with summative iCMA

questions at a deeper level than when they are in formative-only use;

• However there are issues, especially preoccupation with the minutiae of grading;

• Does the answer lie in thresholding or in formative-only use with a ‘carrot’?;

• Exams are a stronger motivator – students decide for themselves to use iCMAs in revision.

How far is it appropriate to go?

• I don’t see online assessment as a panacea;• Some learning outcomes are easier than others to

assess in this way;• Free text questions require students to construct a

response, but there still need to be definite ‘right’ and ‘wrong’ answers (though not necessarily a single right answer);

• However, online assessment provides instantaneous feedback and has been shown to be more accurate than human markers. It can free up human markers for other tasks. It also has huge potential for diagnostic use.

Useful linksPMatch demonstration

https://students.open.ac.uk/openmark/omdemo.pm2009/‘Are you ready for S104?’ (diagnostic quiz, showing a range of question types)

https://students.open.ac.uk/openmark/science.level1ayrf.s104/OpenMark examples site

http://www.open.ac.uk/openmarkexamples/index.shtmlCOLMSCT iCMA initiative

http://www.open.ac.uk/colmsct/projects/sallyjordan

For further information:

• Jordan, S. & Mitchell, T. (2009) E-assessment for learning? The potential of short free-text questions with tailored feedback. British Journal of Educational Technology, 40, 2, 371-385.

• Butcher, P.G. & Jordan, S.E. (2010) A comparison of human and computer marking of short free-text student responses. Computers & Education, 55, 489-499.

Sally JordanStaff Tutor in ScienceThe Open University in the East of EnglandCintra House12 Hills RoadCambridgeCB2 1PFs.e.jordan@open.ac.ukwebsite: http://www.open.ac.uk/science/people/people-profile.php?staff_id=Sally%26%26Jordanblog: http://www.open.ac.uk/blogs/SallyJordan/