Objective assessment

Bob RotheramReader in Assessment, Learning &

Teachingb.rotheram@leedsmet.ac.uk

Intended learning outcomes (Biggs 2003)

write good quality objective test items;

answer (mostly) correctly some questions about today’s subject material;

identify what you need to do next.

Tales of a dissatisfied student

Maths Social work

The importance of…

Assessment Constructive alignment (Biggs

2003) Feedback

Seven pillars of assessment wisdom (Elton 2003) “The drunkard looks under the lamp

post for the keys which he has dropped, not because that is where he has dropped them, but because that is the place where there is light.”

Reliability pursued at expense of validity.

Students should get a profile. Assessments to be valid, reliability

treated within constraints of validity.

Constructive alignment (Biggs) State intended learning outcomes

(ILOs) Find verbs students must perform

to achieve outcomes Assessment tasks require students

to enact the verbs Course activities to be aligned with

enabling achievement of ILOs

Feedback

Potentially very powerful (Brown G, 2001) Best if:

timely perceived as relevant meaningful suggests ways of improvement

(within student’s grasp)

Computer-assisted assessment (CAA): a demo

Social Policy Question Resource (SPQR) 10-question demo

SPQR: some uses Summative

assessment Diagnostic tests Formative tests Informal quizzes Revision aid Convey what

social policy is

Individual questions v. flexible, ‘granular’, ‘learning objects’: insert into lectures gather further info

via web links preparation for

classes …

Resource-based learning (& CAA) best with… Important

background ‘Need to know

before…’ ‘Remedial’ ‘Nice to know’ ‘Much-repeated’ ‘Learned by

doing’

Individual feedback needed

Stuff you don’t like to teach!

Hard to grasp first time

May be needed later, at short notice

(Race 2007)

CAA in your discipline

Use? Potential/advantages? Limitations/disadvantages?

CAA: key points Much ‘up-front’ effort needed Best where:

large numbers of students on introductory courses

questions have long ‘shelf-life’ teams can produce shared item

banks

Objective tests “require a user to choose or provide

a response to a question whose correct answer is predetermined” (Bull & McKenna, 2001)

“only as objective as the test designer makes it” (CAA Centre)

Reputation mixed Best for lower-level skills???

Bloom’s Taxonomy (1956)

knowledge comprehension application analysis synthesis evaluation

Some question types multiple-choice multiple response matching text match

response/fill in blank

extended matching set

‘sore finger’

graphical ‘hotspot’ true/false, yes/no multiple true/false ranking Likert scale numeric assertion/reason essay

Knowledge-testing questions What word means the same as…? What is the most important difference

between…? Which one of the following sequences

shows the correct order of…? What are the major classifications of…? Which method is the most useful for…? What evidence best supports the theory

of…?

Comprehension-testing questions (if not taught or discussed)

Which one of the following is closest in meaning to the term…?

The statement ‘…’ means that… (Various facts are presented.)

Which of the reasons listed below best explains this?

Feedback: minimum statement on whether chosen option

correct or not; (if incorrect) statement giving

correct answer and why correct; clear up possible misunderstanding

about wrong choice; source of info on correct answer; language clear and not

discouraging.

Quality control

Editorial board? Evaluate in use:

Level of difficulty Discrimination – appropriate?

Sources of items

Students Colleagues Staff development events Question-setting workshops Publishers’ websites

Publishers’ websites

E.g. Oxford UP Test Banks, ‘VLE cartridges’

E.g. Bryman, social research methods: 260 MCQs

Check terms of use

Item-writing

See ‘PASS-IT’ Guide, Section 2 Sample MCQs:

‘PASS-IT’ guide, Appendix A

Item-writing exercise

(alone) create 1 MCQ, 1 matching Q, answers, feedback

(2s/3s) ‘shred’ + refine items (whole group) further

consideration of some items

Further issues

Constructing assessments ‘Content-free test’ (Race) Scoring Computerisation, inc.

interoperability Metadata Item banks

1. Who, according to Bob Rotheram, said this: “Assessment is the most important thing we do for HE students.”?

Lewis Elton John Biggs George Brown Sally Brown Phil Race

2. Which of the following statements best reflects John Biggs’ concept of constructive alignment?

A. State the learning outcome we want students to achieve. Find a verb that students need to perform to properly achieve the outcome. Assessment tasks should require students to enact that verb.

B. Don’t just tell students about topics and then give them exams on what we have told them about problem-solving.

C. Constructive alignment is joined-up programme design, making sure that all the components fit together properly.

D. Constructive alignment is used in quality enhancement/assurance in university teaching.

3. Which of the following does Brown G (2001) recommend as components of good feedback on student work?

A. MeaningfulB. Perceived as relevantC. ExtensiveD. Suggesting ways of improvement

(within the student’s grasp)E. TimelyF. Brief

4. Which two levels of Bloom’s Taxonomy is this question most

obviously testing?

5. In the two lists below, match the questions with their types

Multiple-choice Which of these items is the odd one out?

Multiple-response Which of the following best explains…?

Extended matching set Give the proper name for…

Text match response/fill in blank

On this diagram, where is the…?

Graphical hotspot Here are 10 possible answers and 6 questions. For each question, choose the correct answer(s).

Sore finger Which, if any, of these statements is true?

6. In each case below, indicate whether the statement is true or false.

If software is ‘IMS QTI-compliant’, it separates questions from presentation.

Computerised questions need to be IMS QTI-compliant to export them from one CAA package and import them into another.

In 2005, Questionmark Perception did not fully meet the IMS QTI specifications.

Respondus is fully IMS QTI-compliant. The ‘TOIA’ project and ‘JORUM’ repository take IMS

QTI-compliance into account. The ‘RELOAD’ project does not take IMS QTI-

compliance into account. The chance of getting all the answers to this question

correct purely by guessing is one in seven.

Intended learning outcomes (Biggs 2003)

write good quality objective test items;

answer (mostly) correctly some questions about today’s subject material;

identify what you need to do next.

Where next?

Your next steps?

Sources (1) Biggs, J (2003) Teaching for Quality

Learning at University (2nd ed.), Maidenhead, Open University Press.

Bull, J and McKenna, C 2001, Blueprint for Computer-Assisted Assessment: Chapter 3 and Appendix I (Republished 2004 by Routledge Falmer)

Good Practice Guide in Question and Test Design. http://www.pass-it.org.uk/resources/031112-goodpracticeguide-hw.pdf

Sources (2) Brown G, Bull J and Pendlebury M,

(1997) Assessing Student Learning in Higher Education, London, Routledge.

Elton L, (2003) Principles for a fair and honest approach to assessing and representing students’ learning and achievement, HE Academy website.

Race P, (2007) The Lecturer’s Toolkit, (3rd ed.) London, Routledge.