laos session 4: developing quality assessment items (en)

Session 4: Developing Quality Assessment Items

Professor Jim Tognolini

Developing Quality Assessment Items

During this session we will

•define the terms associated with multiple choice items.•discuss the advantages and disadvantages of multiple choice items.•introduce some hints for writing multiple choice items.

Capacity Development Workshop: Test and Item Development and Design, Laos,

September 2016

Parts of a multiple choice itemA load is placed on balance X and then on balance Y.

What is measured by each balance?

Stimulus

Lead sentence,Directions line

Stem or question

Options A,B,D are Distractors

RationalesOptions A,B,D could distract unless student is clear with concept of density, mass and weight

Options

3

Option C is the Key

Options Balance X Balance Y

A Weight Density

B Mass WeightC Weight Mass D Mass Density

Test construction and item analysis workshop, November 2015


September 2016

Parts of a multiple choice item (Continued)

• Directions Line, Lead Sentence - an introduction which directs a student to use the stimulus to answer the item or provides contextual information about the stimulus.

• Stimulus - information required in order to answer the item. Only use stimulus if a student needs the stimulus to answer the question.

• Stem - a question or statement which poses a clearly defined problem that is aligned to the content standard or curriculum benchmark being measured.



September 2016

Parts of a multiple choice item (Continued)

• Options - the answer choices for students to select when answering the question.

• Key - the correct answer.

• Distractors - the incorrect answers.

• Rationales - justifications that explain why a certain distractor is plausible, yet incorrect, or demonstrates a common misconception.



September 2016

Quick quiz (True or False or Don’t Know [Need More Information])

1. One function of the lead sentence is to direct students to use the stimulus to answer the item.

2. Item rationales explain why distractors are plausible.

3. Distractors are incorrect answer options.

4. A stimulus should be included on an item even if it is not necessary to answer the item.

5. Answer options include the key (correct answer) and the distractors.

6. Reading passages, maps, diagrams, and tables are examples of stimuli.



September 2016

Item types

Answer Type

Correct Answer

Best Answer

Negative Answer

Stem Type

Direct Question A B C

Sentence Completion D E F

Picture/Diagram G H I



September 2016

Pollination is best defined as the

a) growth of the pollen tube in the ovule.

b) germination of pollen grains.

c) carrying of pollen by insects from flower to flower.

d) transfer of pollen from anther to stigma.

Direct question and best answer type (Type B)



September 2016

Requirements of writing good items

As test developers, you must have

• a thorough knowledge of the subject matter - including knowledge of popular fallacies and misconceptions;

• an understanding of the content standards of the curriculum and the performance of individuals;

• good written communication skills;

• technical item writing skills; and,

• imagination and ingenuity.



September 2016

Sources of item ideasSources of item ideas include• chance ideas and inspirations; • the work (verbal, written) of students;• the items and ideas of other teachers;• understanding of what an expert in this area is expected to know/

be able to do; and,• imagination and ingenuity.



September 2016

Advantages of multiple choice items

Advantages include

• versatility (adaptable for various levels of learning outcome including, simple recall of knowledge, analysis of phenomena, application of principles, interpreting cause and effect relationships, etc.) ;

• increased validity (more questions therefore greater coverage of the syllabus);

• increased reliability because of objectivity of marking; and,

• increased efficiency (easily marked).



September 2016

Disadvantages of multiple choice items

Disadvantages include

• not adaptable to measuring certain learning objectives including articulating an explanation, displaying thought processes, etc.;

• susceptibility to guessing;

• difficulty of construction.



September 2016

A Couple of myths associated with multiple choice items

• Multiple choice items can only be used to measure lower-level outcomes such as those based on knowledge, facts and principles.

• Individuals can perform well simply by guessing.



September 2016

Hints for writing multiple choice items

Align the item with the learning outcome.

Business Studies Objective: Compare and contrast different forms of business organizations (Public Company, Private Company, Proprietorship, Partnership ) Which of the following forms of ownership is the outcome of a voluntary agreement? a) Public Companyb) Partnership*c) Proprietorshipd) Private Company

Is this assessing the learning outcome?



September 2016


Each item should focus on an important concept.

The early development of analysis of variance (ANOVA) was mainly due to work done by Sir Ronald A. Fischer.

What was Fischer’s middle name?

a) Alan.b) Albert.c) Aylmer.d) Arthur.

Is this assessing an important learning objective?



September 2016


The language is simple, clear and unambiguous.

Is this clear and concise language?

The local community theatre group in Jakarta is performing a play in the Local Public School auditorium. There are 15 rows of seats. Each row contains 28 seats.

What is the largest number of tickets that the communitytheatre group can sell to fill the auditorium for one performance?

a) 400 tickets.b) 420 tickets*.c) 450 tickets.d) 470 tickets.



September 2016


How has this stem improved?

A theatre has 15 rows and each row contains 28 seats. What is the largest number of tickets that the theatre can sell to fill the auditorium for one performance?

a) 400 tickets.b) 420 tickets*.c) 450 tickets.d) 470 tickets.

Better …



September 2016


The stem and answer options must be grammatically consistent with one another.

What is the answer? Has the question really assessed the learning outcome?

The type of vessel that carries blood from the heart to the lungs is an

a)artery.b)capillary.c)node.d)vein.



September 2016


How has the item improved?

The type of vessel that carries blood from the heart to the lungs is

a)an artery*.b)a capillary.c)a node.d)a vein.

Better …



September 2016


The stem and distractors must not give clues to the key.

What is the answer? Has the question really assessed the learning outcome?

Seetha is writing a report on different types of ecosystems. In which source would she most likely find this information?

a) Cooking with Native Plants.b) A Guide to Rock Collecting.c) Ecosystems of Asia* .d) The Big Book of Animal Habits.



September 2016


The item presents a single clearly formulated question in the stem and is written in question or sentence completion format.

Can you cover the options and still answer to the item?

From the article, the reader can tell that

a)turtles like to hide under the rocks.b)dogs like to play in the snow.c)squirrels like to eat mulberries. d)cats like to chase mice.



September 2016



From the article, the reader can tell that turtles like to

a) hide under the rocks.b) play in the snow.c) eat mulberries. d) chase mice.

Better …



September 2016


The stem and answer options are phrased in positive terms.

Have you ever been asked what is not the answer to a question?

None of the following cities are state capitals except a) Bellary.b) Hyderabad*.c) Katni.d) Pune.



September 2016



Better …

Which one of the following cities is a state capital? a) Bellary.b) Hyderabad*.

c) Katni.d) Pune.



September 2016


There are no repetitions in the options.

Why do you think that repetitions in the options can lower validity?

Milk can be pasteurized at home by

a)heating it to a temperature of 33oC for 30 minutes.b)heating it to a temperature of 43oC for 30 minutes.c)heating it to a temperature of 53oC for 30 minutes.d)heating it to a temperature of 63oC for 30 minutes*.



September 2016


Better …

The minimum temperature that can be used to pasteurise milk at home is

a) 33oC.b) 43oC.c) 53oC.d) 63oC*.




September 2016


Ensure that there is only one correct or clearly best answer.

What is the correct answer to this item?

Which one is the odd one out?

a) Billiards.b) Cricket.c) Hockey.d) Football.



September 2016


Ensure that there are neither repetitions nor opposites in the options.

Which would be the most likely effect of this change in fiscal policy?

a)The inflation rate would decline.b)The unemployment rate would rise.c)Consumer spending would increase. *d)Consumer spending would decrease.




September 2016


Answer options are plausible and similar in context, ideas, focus, phrasing and length.


Why has the item been made d) so long?

Epistemology is the branch of philosophy dealing with

a) the nature of science.b) morality.c) beauty.d) the nature and origin of knowledge – that is, the manner in which

human beings sense and process external stimuli in the form of knowledge.



September 2016


Specific determiners (always, all, never, only, none) must be used cautiously.

What is the correct answer to this item? How could the weak

students get this item correct without really knowing the answer?

Achievement tests help students to improve their learning by

a) encouraging them all to study hard.b) informing them of their progress.c) giving them all a feeling of success.d) preventing any of them from neglecting their work.



September 2016

Hints for writing multiple choice itemsUse “All of the above” and “None of these”

sparingly.


What is wrong with this item?

Which of the following levels are included in Bloom’s Taxonomy?

a) Comprehension.b) Application.c) Analysis.d) All of the above.



September 2016


What is the problem with using “None of the above” as the answer?

Which one of the following is a level in Bloom’s Taxonomy for the cognitive domain?

a)Critical Thinking.b)Scientific Thinking.c)Reasoning.d)None of the above.

Another Example...



September 2016


Ensure that answer options do not overlap with each other.


What is wrong with this item?

If the scores on a test have a reliability of 0.75, what percentage of an observed score is attributable to errors of measurement?

a) Over 5%.b) Over 10%.c) Over 20%.d) Over 30%.



September 2016



Better …

If the scores on a test have a reliability of 0.75, what percentage of an observed score is attributable to errors of measurement?

a) 2.5%.b) 5%.c) 25%.d) 50%.



September 2016

Writing multiple choice items from an editorial point of view

The items must be • free from spelling, punctuation, grammatical and other editorial

faults;

• presented with appropriate text fonts (size, type), highlighting (bold, underlining, italics) and layout (paragraphing and positioning);

• arranged in such a way that the students do not have to turn pages to link sources to questions (stems to options); and,

• arranged in the paper in the order from easiest to hardest.



September 2016

One result of having a theatre in a community is more

a) jobs.b) teachers.c) stores.d) crime.

Quick Quiz What is the Problem with Item 1?



September 2016

In Charles Dickens’ novel A Christmas Carol, which characteristic describes Ebenezer Scrooge?

a) Miserly. b) Nervous. c) Inquisitive. d) All of the above.




September 2016

The table below shows snowfall totals for Pokharan in February.

Based on the table, what percent of days in February had

snowfall of 5 cm or more?

a) Less than 20%.b) Less than 30%.c) More than 65%.d) More than 75%.




September 2016

Why should candy be eaten sparingly between meals?

a)Candy depletes energy.b)Candy causes diabetes. c)Candy causes headaches.d)Candy dulls the appetite for other foods essential for proper nutrition.




September 2016

Quick Quiz (True or False or Don’t Know [Need More Information])

A GOOD ITEM

1. Measures a specific learning objective.

2.Contains subject matter and vocabulary that is above the student’s grade-level.

3. Has only one correct answer or clearly best answer.

4. Assesses trivial or obscure subject matter.

5. Free from grammatical clueing.

6. Is free of negative wording such as “not” or “none of the above.”



September 2016


A GOOD ITEM

7. Assesses more than one concept.

8. Contains options that are opposite of one another.

9. Contains distractors that assess common errors or misconceptions.



September 2016


1. Distractors should be parallel in content, structure, and length.

2. Cognitive level refers to the difficulty level of an item.

3. Items should be written so that the content in the item is accessible to the widest range of students.

4. Parallelism refers to when a student from different ethnic, sex, or cultural groups perform differently on an item.

5. Controversial items are often assessed with the multiple-choice format.



September 2016


6. Item fairness means that the item assesses all students at the appropriate age and enrolled grade-level.

7. An example of bias is presenting a culturally stereotypical situation in the item.



September 2016

laos session 4: developing quality assessment items (en)

Education