Assessment and feedback: technology to the rescue?

Bob RotheramNational Teaching Fellow

Outline ‘Objective testing’

‘Social Policy Question Resource’ (SPQR) Digital audio for assessment feedback

‘Sounds Good’ With each:

What to use technology for How to do it Limitations and obstacles

Objective testing “Objective tests require a user to choose

or provide a response to a question whose answer is pre-determined … the marking of the responses is completely non-subjective because no judgement has to be made on the correctness or otherwise of an answer at the time of marking. … An objective test is only as objective as the test designer makes it.” (Bull & McKenna 2004)

Some question types

multiple-choice (MCQ) multiple response text match response/fill in the blank true/false, yes/no Likert scale numeric essay

Objective testing: objections?

Well?

MCQ: easy?

MCQ: feedback

Feedback: recommendations statement on whether chosen

option correct or not (if incorrect) statement giving

correct answer and why correct clear up possible misunderstanding

about wrong choice source of info on correct answer language clear and not discouraging

Bloom’s Taxonomy (1956)

knowledge comprehension application analysis synthesis evaluation

Knowledge questions What word means the same as…? What is the most important difference

between…? Which one of the following sequences

shows the correct order of…? What are the major classifications of…? Which method is the most useful for…? What evidence best supports the theory

of…?

Comprehension questions

Which one of the following is closest in meaning to the term…?

The statement ‘…’ means that … [Various facts are presented.]

Which of the reasons listed below best explains this?

Objective testing: some uses Summative

assessment Diagnostic tests Formative tests Informal quizzes Revision aid Subject intro

Individual questions v. flexible, ‘granular’, ‘learning objects’: insert into lectures gather further info

via web links preparation for

classes …

Objective testing:limitations, obstacles

Other assessment also needed Up-front effort required Question-checking Maintenance, shelf-life Collaboration Data portability

Writing questions

Bull & McKenna, (2004), Blueprint for Computer-Assisted Assessment

PASS-IT (2004), Good Practice Guide in Question and Test Design

In your discipline?

Use? Potential/advantages? Limitations/disadvantages?

Sounds Good ‘Quicker, better assessment using

audio feedback’ Coursework:

formative, summative, individual, group Leeds Met (2008) Leeds Met, Newman, Northampton,

York St John (2008-9) [Subject centres: Engineering, GEES]

Sounds Good: some numbers

£35k + £15k (JISC) 38 lecturers 1200+ students Cohort sizes: 3 to 151 Levels: 1 to doctoral 37 presentations!

Marian Tuck’s coursework

Book review length, tips assessment criteria

Read it Audio feedback

listen (as Marian Tuck) how did it feel? other comments?

Tools, techniques

Digital audio feedback MP3 recorders ‘Audacity’ software ‘WIMBA’ voice tools

Audio files via: Email VLE

Early comments

(Student) “Very helpful. It felt like the tutor was able to expand more… . Often when you read feedback, things can get misunderstood or meant in a different way. I felt this way was very clear.”

(Staff) “I think with practice this will get quicker as I get more used to things.”

More staff opinions Staff like audio feedback:

Quality, quantity… “I was able to give … more detailed

and pertinent feedback … [It] became almost an online tutorial.”

“An ideal medium to assist in the development of skills and confidence of students.”

Student opinions

Students like audio feedback! personal, detail, careful consideration

“Very helpful. It felt like the tutor was able to expand more… . Often when you read feedback, things can get misunderstood or meant in a different way. I felt this way was very clear.”

On the other hand…

Some students prefer written feedback Should staff oblige?

Effort to produce, student ‘skimming’…

Some want audio + written Whether/how to do it?

Audio feedback:Practice guidelines Handheld recorder more convenient?

record direct to MP3; USB port Keep files short (<5 mins?) ‘Good enough’ sound quality (32kbps

mono?) Get approval for audio use

See www.soundsgood.org.uk

Time saving? Don’t expect to save time

immediately Time to send audio files

Problem if many students Best chance of saving staff time if:

give lots of feedback write slowly but record speech quickly comfortable with technology easy to send audio feedback

Is audio feedback worth it?

Experienced practitioner Better, but may not be quicker Give an extended trial Worth it for some assessment, esp.

if detail, personal touch valuable to groups

Other uses of audio!

Technology to the rescue?

Not a magic bullet Use appropriately

In favourable circumstances Make informed choices on methods

Technology can bite! “To err is human, but to really screw

things up requires a computer.” (Anon)

Sources (1)

Bull J & McKenna C (2004), Blueprint for Computer-Assisted Assessment, RoutledgeFalmer, London

PASS-IT (2004), Good Practice Guide in Question and Test Design, http://staffnet.kingston.ac.uk/~ku36708/RRR/docs/goodpracticeguide.pdf

Sources (2)

Rotheram B, (2009) Practice tips on using digital audio for assessment feedback, http://sites.google.com/site/soundsgooduk/downloads/Audio_feedback_tips_3.pdf