Multiple Choice Item ConstructionD. Michele JacobsenAssistant Professordmjacobs@acs.ucalgary.ca

Test Development Process1. Statement of Goals2. Content Outline3. Table of Specifications4. Item Selection5. Item Construction6. Composition of Instructions7. Development of Answer Sheets8. Construction of Answer Keys9. Test Administration10. Test Revision

Todays Focus3. Table of Specifications4. Item Selection5. Item Construction

Two Important Characteristics of TestsReliabilityconsistencyfree from extraneous sources of errorValidityhow well a test measures what it is supposed to measure

Formative vs. Summative TestsFormativemonitor progress toward goals within a course of studySummativeassess overall achievement of course goals

A.Table of SpecificationsBlue print for testPurposeensure proper emphasis given to all elements of a course of studycontent validityGuide for writing items

Table of Specifications (2)Start with Instructional Objectives Biology 30: Circulation Unit Test1. Identifies parts of circulation system.

2. Explains function of parts in relation to whole system.

3. Distinguishes between circulation system and other bodily systems.

Biology 30: Circulation

Levels

Of

Understanding

Content

Knowledge

Comprehension

Application

Parts

Function

Relation to other systems

Need to Consider and Decide:Length of TestWeight to be given to each objectiveWeight to be given to each level of taxonomyEstimate number of items in each cell

Biology 30: Circulation

Levels

Of

Understanding

Content

Knowledge

Comprehension

Application

Parts

10

5

0

15

Function

5

5

5

15

Relation to other systems

0

5

10

15

15

15

15

45

B.Item SelectionTypes of ItemsObjectively Scored (Selection)true/falsecompletionmatchingmultiple choiceSubjectively Scored (Supply)interpretive exercisesessay

C. Item ConstructionSelected Response Test ItemsItem construction skills for valid and reliable measures of student achievement.Guidelines which apply to all typesSpecific Suggestions for writing each typeAdvantages and Disadvantages of Each

Guidelines for Writing Objective Items1.Construct at appropriate level of difficulty for examinees

2.Include Items at appropriate level of difficulty for purpose of test.

3.Test significant elements of a course.

4.Write independent items.

5.Construct questions free from extraneous reasons for problems.

6.Communicate the question in clear, concise language.

7.In the correct alternatives, paraphrase statements from the text.

8.Exclude clues to correct answer.II. Guidelines for Writing Objective Items

III. Guidelines for Writing Objective Items9.Provide one correct answer.

10.Edit the Items.

Alternate Response ItemsInvolves the selection of one of two alternativestrue / falseyes / noright / wrongfact / opinion Mainly for Knowledge & ComprehensionCan be written at higher levels

True / FalseWord statements clearly. Vague or ambiguous wording will confuse students.Avoid overgeneralizing.Poor:Heavy smoking causes lung cancer.T FBetter:Heavy smoking often causes lung cancer. T FAvoid Trick questions.i.e., General Wolseley led Canadian troops to Manitoba in 1870.3 tricks:Colonel, British, Not yet a provinceDo not use trivial statements to pad out the number of questions and marks to arrive at a predetermined level.

True / FalseStatements should be entirely true, or entirely false: Unacceptable:In King Lear, Regan ordered Gloucesters eyes to be plucked out and Gloucester died when he jumped off the cliff of Dover. Acceptable:In King Lear, Regan ordered Gloucesters eyes to be plucked out. (T)In King Lear, Gloucester died when he jumped off the cliff at Dover. (F)

True / FalseAvoid using universal descriptors such as never, none, always, and all. Testwise students will recognize that there are few absolutes. Avoid negative words, as they are often overlooked by students.Poor:It was not unheard of for Henry VIII to close monasteries in England.T FBetter:Henry VIII closed some monasteries in England. (T)

True / FalseDo not include two ideas in one statement unless you are evaluating students understanding of cause and effect relationships. Poor:Porpoises are able to communicate because they are mammals.T FBetter:Porpoises are mammals.T F Porpoises are able to communicate.T F Provide a T and F beside each statement and ask students to circle correct answer. Avoids problem of students writing illegible letters.

True / False & VariationsInclude more false than true statments in any given test and vary the number of false statements from test to test. tendency to mark more statements true than false.discrimination between those who know the content and those who do not is greater for false expressions. Avoid using negative statements.Under the demands of the testing situation, students may fail to see the negative qualifier.

Matching ItemsConsist of a column of premisesa column of responsesdirections for matching the two.Similar to multiple choice, but easier and more efficient to constructCan be written to assess Knowledge, Comprehension, Application, Analysis level behaviors

Guidelines for Writing Matching ItemsProvide clear instructions on how to indicate the correct answers.Indicate whether the same response can be used more than once.Maintain grammatical consistency within and between columns.within a column: either sentence or point formbetween columns: one or the otherEnsure that any matching question appears entirely on one page.

2. Guidelines for Writing Matching ItemsProvide an unequal number of premises and responsesreduces guessing and eliminationincreases measure of comprehension Avoid designing questions which require students to draw lines between premise and response. confusing for student and markerprovide space for letter or number answers

3. Guidelines for Writing Matching ItemsMake sure lists are homogeneous.i.e., do not include items testing names, dates, and events.Instead, make every response plausible Make the wording of the premises longer than the wording of the responses. Identify the items in one list with numbers and those in the second list with letters.

Example:Directions:1. On the line to the right of each phrase in Column I, write the letter for the word in Column II that best matches the phrase.2. Each word in Column II may be used once, more than once, or not at all.Column IColumn II1. Name of the answer in addition problems.2. Name of the answer in subtraction problems.3. Name of the answer in multiplication problems.4. Name of the answer in division problems.A. DifferenceB. DividendC. MultiplicandD. ProductE. QuotientF. SubtrahendG. Sum

Short Answer Test ItemsTypically, the student is asked to reply with a word, phrase, name, or sentence, rather than a more extended response.Direct Questions / Short AnswerWho is the current Prime Minister of Canada?Incomplete Sentences / Fill In the BlanksThe current Prime Minister of Canada is _____? Items are fairly easy to construct and markAssess mainly knowledge, comprehension, and some application.

Guidelines for Writing Short Answer ItemsQuestions must be carefully worded so that all students understand the specific nature of the question asked and the answer required. Poor: Wellington defeated Napoleon in _____ ? Better: In what battle fought in 1815 did Wellington defeat Napoleon? OR In what year did Wellington defeat Napoleon at Waterloo?

II. Guidelines for Writing Short Answer ItemsWord completion or fillin questions so that missing information is at, or near the end of, the sentence. Makes reading and responding easier. Poor: In the year ______ , Canada turned 100 years old. Better: Canada turned 100 years old in the year _____ . Instructions and teachers expectations about filling in blanks should be made clear. Indicate whether each blank of equal length represents one word or several words, whether long blanks require sentences or phrases, and whether synonymous terms are accepted.

III. Guidelines for Writing Short Answer ItemsWhen an answer is to be expressed in numerical units, the unit should be stated. Poor: If a room measures 7 metres by 4 metres, the perimeter is _____ .Better: If a room measures 7 metres by 4 metres, the perimeter is _____ metres (or m). Do not use too many blanks in completion items. The emphasis should be on knowledge and comprehension, not mind reading! Consider: In the year _____ , Prime Minister _________ signed the __________ , which led to a ___________ which was __________ .

Multiple Choice ItemsInterpretive ExerciseGuidelines for WritingAdvantages & Disadvantages

1. The capital city of Canada is a. Vancouver b. Montreal c. Toronto*d. OttawaTerminology: Multiple Choice1. Stem: presents the problem2. Keyed Response: correct or best answer3. Distracters: appear to be reasonable answers to the examinee who does not know the content4. Options: include the distracters and the keyed response.

AmbiguityExtrinsicDesirable quality in multiple choice itemsOutside the itemAllows discrimination between those who know material and those who do not.IntrinsicUndesirableShould be avoidedInside the itempoor wordingmore than one answerEven those who know content have difficulty choosing correct answer

Types of Multiple Choice ItemsCorrect Answer*Only one correct responseBest Answerrequires examinee to select alternative closest to being correctfine distinctionsMultiple AnswerMore than one correct or best answer

Interpretive ExerciseUsually begins with verbal, tabular or graphic information which is the basis for 1 or more multiple choice questions.map, passage from a story, a poem, a cartoonCan challenge students at various levels of understandingapplication, analysis, synthesis, evaluationExercise contains all information needed to answer questionsReadily adaptive to the more important outcomes of disciplines.

Interpretive Exercises (cont)ExamplesIf student answers incorrectly it is because they have not mastered the thinking or reasoning required by the question, NOT because they failed to memorize background information.Math questions: give students the formulas, test ability to apply concepts, rather than ability to memorize formulas.

Guidelines for Writing: Multiple Choice ItemsState stem in the form of a question.WeakCanada isa)a countryb)where you livec)between the Pacific and Atlantic oceansd)all of the above BetterBetween which two oceans is Canada located?a)Atlantic and Indianb)Atlantic and Pacificc)Pacific and Indian

Guidelines for Writing: StemsPlace most of the subject matter in the Stemensures full statement of problemEliminate extraneous material from the Stemgoal is to measure student achievement, not to present new materialmaximize use of time for demonstrating understanding, not reading abilityAvoid Negatively phrased Stemsstudents may miss the qualifieruse only when learning outcome requires this type of differentiation

I. Guidelines for Writing: MCEnsure similarity among alternatives with regard to:grammatical structurelengthmode of expressionGrammatical errors provide unintentional clues to the answerWhen in doubt, students will select the longest alternative as the correct answer

Example: Length of AlternativesNeurotics are more likely than psychotics to

a.be dangerous to societyb.have delusional symptomsc.be dangerous to themselves*d.have insight into their own inappropriate behavior but nevertheless feel rather helpless in terms of dealing with their difficulties

II. Guidelines for Writing: MCMake one of the alternatives the most clearly correct or best answerexception: multiple answer formreduces intrinsic ambiguityreduces frustration during test Make distracters plausibledesire to attract students who really do NOT know the answer to the questioncreate distracters from elements of the correct responseimproves reliability of item

Example: Undemanding DistractersMickey Mouses two nephews are

a) Huey, Dewey and Louieb) Clarabelle Cow*c) Morty and Ferdyd) Abbott and Costello

The Role of Humour?

III. Guidelines for Writing: MCAvoid parallel language between the Stem and the Correct Responsegives clues to keyed responseemphasizes testwiseness, not knowledge Randomly distribute answers across the alternative positionsinexperienced test writers emphasize b and c alternatives (hide the answer!!)do NOT use an interpretable order of keyed responses

IV. Guidelines for Writing: MCUse qualifiers such as all of the above and none of the above sparinglytestwise students will use process of elimination to select answerdo NOT use to pad out the distracters because you cannot think of another one.

Advantages of Multiple Choice Itemsallow more adequate sampling of content.tend to more effectively structure the problem to be addresseditems can be more efficiently and reliably scored than supply itemsdifferent response alternatives can provide diagnostic feedback (item analysis) items can be constructed to address various levels of cognitive complexity

Disadvantages of Mult.Choice Itemsdifficult & time consuming to construct good itemsleads to emphasis on other selected response item typescan lead the instructor to favour simple recall of factshigh degree of dependence on students reading and instructors writing abilitycan be difficult to achieve clarity of expressionmeasuring synthesis and evaluation can be difficultinappropriate for measuring outcomes that require skilled performance

Wrap-Up