chapter eleven: improving teacher-developed · pdf filedifficulty index (level)? • often...
TRANSCRIPT
Chapter Eleven: Improving Teacher-Developed
Assessments
Danielle LamarreDarcy ChurchJason HowardSarah Fearon
What are three sources of test-improvement judgments?
• Your own assessment instruments• Colleague judgments• Judgments of students
3 sources test-improvement judgments
Yourself• Adherence to item-specific guidelines and general item
writing commandments• Contribution to score-based inference• Accuracy of content• Absence of content lacunae• Fairness
Your colleagues• Colleagues review assessments based on the previously
discussed criteriaYour students
• Simply have the students complete the assessment first and then distribute a questionnaire; conduct this consecutively not simultaneously!
What is item difficulty?
The measurement of the percentage or proportion of test takers who answer the item correctly
How do you compute item difficulty index (level)?
• Often referred to as “p value” P = R/T• R= the number of students responding correctly
(right) to an item• T= the total number of students responding to the
item• P value can range from 0 to 1.00• For example a p value of .98 would signify an
item answered correctly by almost all students, Similarly, an item with a p value of 0.15 would be one that most students (85%) missed
What is the item discrimination index?
• How frequently an item is answered correctly by those who perform well on the total test
• Fundamentally, an item-discrimination index reflects the relationship between students’ responses for the total test and their responses to a particular item (page 278, 2014)
ITEM-DISCRIMINATION INDICES
Type of item Proportion of correct responses on total test
Positive Discriminator High scorers>Low scorers
Negative discriminator High scorers<Low scorers
Nondiscriminator High scorers=Low scorers
How do you compute the item discrimination index?
Procedure:1. Order the test papers from high to low score by total score
2. Divide the papers into a high group and a low group, with an equal number of papers in each group
3. Calculate a p value for each of the high and low groups
4. Subtract pl from ph to obtain each item’s discrimination index (D)
GUIDELINES FOR EVALUATION THE DISCRIMINATING EFFICIENCY OF ITEMS
Discrimination Index Item Evaluation0.40+ Very good items0.30 – 0.39 Reasonably good items, but
possibly subject to improvement
0.20 – 0.29 Marginal items, usually needing improvement
0 – 0.19 Poor items, to be rejected or improved by revision
Example • If 75% of the students in the top group answer an
item correctly, and 20% of the students in the bottom group answered the item correctly, what is the item discrimination index?
• D = 0.75 – 0.20 = 0.55
• Is this a good item?
• This is a good item
What is a distractor analysis?• Analyzing the wrong answers or distractors• Scrutiny of a student’s empirical performances
on the different answer options contained in multiple-choice and matching items
• Good distractors are highly effective with the lower performing group
What are the five review criteria teachers consider to improve assessment
procedures?
1. Adherence to item-specific guidelines and general item-writing commandments.
a) Follow the five general item-writing commandments listed in Chapter 6
b) Follow the item-specific guidelinesc) Fix any violations
2. Contribution to score-based inference.
What are the five review criteria teachers consider to improve assessment
procedures?
3. Accuracy of content. Be sure that the content of the assessment is accurate and relevant.
4. Absence of content lacunae. Make sure that all key elements of the content are present.
5. Fairness. Ensure that the assessment does not partake in any bias. Perform a bias review as a necessary precaution.
Reference
Popham, W.J. (2014). Classroom assessment: What teachers need to know (5th ed.). Boston, MA: Pearson Education Inc.