{

Quality of Medical Examination

QuestionsDr. Sanjoy SanyalAssociate Dean and ProfessorTexila American University, Guyana

Scramble the questions (either manually or through exam software)

Number of questions vis-à-vis chapter length

Number of questions vis-à-vis chapter importance

Standard Format (Heading, Font size, Style, Pagination etc)

Signed by Faculty, HoD, Chair of Examination Quality Control, Dean

General Aspects

True-False MCQ Family

Good

True-False MCQ Family

Options 1, 2, 3 cannot be judged as absolutely true or false

A group of experts will not agree with the answer

Only option 4 is unambiguous and clearly true

Flawed

True-False MCQ Family

‘May’ may be true ‘Usually’ is harder to define Most experts will not agree whether 4th option is true or false

Flawe

d

True-False MCQ Family

Each of these options may or may not be associated with VSD

So, which is / are the correct answer(s)?????

Flawed

Either Stem is unclearAnd / or Options contain vague terms

And / or Options are partially correct

A group of experts will have difficulty in arriving at a consensus on the correct answer

Flawed True / False MCQ

T/F MCQ on a Lighter Note!!!

It all depends on individual perspective

Parts of an A-type MCQ Item (1-Best Answer (OBA))

Near-perfe

ct 1-

Best Answ

er MCQ

ItemThe whole thing is called an

Item

Previous OBA MCQ Analysis

Note, Stem is long, Options are short (next slide) This is a Front-loaded Question Item Lead-in Question asks ‘most likely’ diagnosis Other options are not totally wrong It should be possible to diagram all the options in

a single continuum from ‘most likely’ to ‘least likely’

This is called Uni-dimensional Question Item

Item Shape

Flawed OBA MCQ

Inherently flawed: Long Options, Verbs in Options, Small Stem

It is a Back-loaded Question Item Too many adjectives – Which is ‘more true’ than the other? Trying to compare apples with oranges Options are heterogenous and Multi-dimensional Options cannot be rank-ordered in a single dimension Item cannot be answered without looking at the options

Highly

flawed

T/F vs. OBA MCQTFA Easy to create Very few items in

the real-world are absolutely true or false

Correct answer depends on individual perspective

Tests recall of isolated fact - Knowledge

OBA More difficult and time-

consuming to create Most commonly used When structured in a

linear rank-ordered fashion, from most likely to least likely, there is no ambiguity in the correct option

Tests higher levels of Bloom’s Cognitive Domain – Comprehension, Application, Analysis (next

OBA MCQ Based on Cognitive Domain of Bloom’s Taxonomy

OBA MCQ Based on Cognitive Domain of Bloom’s Taxonomy

1. Grammatical Errors

Options A and C do not follow grammatically or logically from the Stem

2. Logical Errors

Stem too short Options too long Options A, B, C are collectively exhaustive and include all

possibilities Options D and E are partially correct and confusing fillers Options, D, E cannot be rank-ordered in the same

dimension as Options A, B, C

Back-loaded question

3. Absolute Terminology

Stem is inconclusive and un-focused Options

Are long Contain verbs Are Multi-dimensional (cannot be rank-ordered in a

Single-dimension) Contain Absolute terms (Test-wise student will

immediately eliminate options C and D)

4. Long Correct Answer

Option C is longer than others It contains double options Examiners tend to write the correct answers first,

and make them long to ensure they are comprehensive and correct

Test-wise student immediately notices that

5. Word Repeats

Types of Word-repeats Actual: ‘Unreal’ in Stem and ‘Derealization’ in

Option Metaphorical: The word ‘Bone’ in Stem, and

the pre-fix ‘Osteo’ in the Option Both serve as cues to the test-wise student

6. Options Long, Complicated

Stem may be long or short – depending on purpose

Here Stem is superfluously long Options are too long, complicated, double Tests reading speed rather than Cognition

7. Inconsistent Numeric Data

Options A, B, C are ranges; D, E are specific figures

Should be either ranges or figures; not mixed Option C range includes options D and E Test-wise student will eliminate D and E Such Question Items test recall of facts

rather than understanding of concepts

8. Vague – Poorly Constructed

Back-loaded question Vague frequency terms in Options – ‘Usually’, ‘Often’

Multi-dimensional

9. Poor Item Construction

Correct way to write the Options in this example:A. No follow up was made of non-vaccinated childrenB. Number of cases was too smallC. The trial involved only boysD. Write a new option D

Language is not parallel

Mismatch between Stem and Options

Options are in illogical order

10. ‘None of the Above’

11. Unnecessarily Complicated

Summary of Technical Item Flaws

Summary of Technical Item Flaws

Item construction in General Focus on Important Concepts Don’t waste time on trivial facts Avoid ‘tricky’ or overly complex ItemsIssues Pertaining to Stem Maximum information should be in the Stem Should be long, in general But, avoid superfluous information Should be answerable without looking at the Options

Avoid Negative terms – ‘Except’, ‘Not’

Guidelines for Item Construction

Issues Pertaining to Options Options should be short Should grammatically consistent and logically

compatible with Stem Avoid Absolute terms – ‘Always’, ‘Never’,

‘All’ Avoid – ‘All of the above’, ‘None of the

above’ Avoid Vague terms – ‘Usually’, ‘Frequently’ Distractors should be plausible Distractors should be same relative length

as correct Option List Options in logical or alphabetical order

Guidelines for Item Construction

Stem: Focused yet sufficiently comprehensive Student should be able to arrive at the answer

without looking at the options Lead-in:

In the form of a question Keep verbs in the Lead-in question

Options: Short Approximately equal length Uniform between each other Uni-dimensional Capable of being rank-ordered from ‘Most likely’ to

‘Least likely’

Do’s of OBA MCQs

Stem: Too short, inconclusive, illogical, grammatically flawed, contradictory

No Stem at all Lead-in

In the form of a half-sentence or statement Not in the form of a question

Options: Long and / or complex and / or contains verbs Contains Absolute terms – ‘Always’, ‘Never’ Unequal length Non-uniform between each other Multi-dimensional Cannot be rank-ordered from ‘Most likely’ to ‘Least

likely’ Shares a common word from the Stem

Don’ts of OBA MCQs

In Stem or Lead-in Each of the following is correct EXCEPT Which of the following statements is NOT

correct? Which of the following statements is

CORRECT? In the Options

All of the above None of the above A and C above B and D above

Other Don’ts in OBA MCQs

Basic Rules – Revisited

Basic Rules – Revisited

Options

Thank you for watching