1. Creating Valid & Reliable Assessment Items Greg Sherman, Ph.D. Instructional Design and Evaluation

2. Strategies for effective selected and constructed response. Rubrics Portfolios Improving Instructional Practice Projects #1 & #2 3. Better Better = Teaching Assessment 4. We become just by performing just acts. -Aristotles Nicomachean Ethics 5. Learning = Doing 6. Assessment: Formally measuring status of students with respect to educational variables of interest. 7. Educational variables of interest Expected Learning Outcome Performance Unexpected Learning Outcome Performance Embedded Practice Performance Expected Attitudes & Motivation Unexpected Attitudes & Motivation Instructional Component Perception Social Interaction Concerns 8. Formal Assessment [Teacher] 9. Quality Assessments: Reliable Valid Free from Bias 10. Three Easy Steps for Developing Valid & Reliable Assessment Items 11. 1. Identify and Classify Performance Outcomes 2. Select most appropriate TYPE of assessment item 3. Follow the rules for writing good items 12. Step OneVICAM 13. Step Two Selected Response Multiple Choice (single or multiple answers) Matching Binary Choice (i.e. True/False) Multiple Binary Choice OrderingConstructed Response Fill-in-the-Blank (including math problems) Labeling (unless a list of choices is provided) Short Answer (including math problems) Essay Project Performance 14. Core Subject [Social Studies] Standard 1.2 The student will describe the stories of American leaders and their contributions to our country, with emphasis on George Washington, Benjamin Franklin, Abraham Lincoln, and George Washington Carver. 15. Verbal Information Valid Type(s) of Constructed Response Fill-in-the-Blank Labeling Short Answer Essay Project PerformanceValid Type(s) of Selected Response Multiple Choice Matching True/False Ordering 16. The U.S. president who was most responsible for helping African American slaves become free men and women was __________. 17. IS: Concrete Concept 18. Core Subject [Social Studies] Standard K.9The student will recognize the American flag, the Pledge of Allegiance, and that the President is the leader of the United States. 19. IS: Concrete Concept Valid Type(s) of Constructed Response Fill-in-the-Blank Labeling Short Answer Essay Project PerformanceValid Type(s) of Selected Response Multiple Choice Matching True/False Ordering 20. Circle the official flag of the United States of America: 21. IS: Defined Concept 22. Core Subject [Social Studies] Standard 3.9 The student will identify examples of making an economic choice and will explain the idea of opportunity cost (what is given up when making a choice). 23. IS: Defined Concept Valid Type(s) of Constructed ResponseFill-in-the-Blank Labeling Short Answer Essay Project PerformanceValid Type(s) of Selected Response Multiple Choice Matching True/False Ordering 24. Paul just moved into a new apartment. He has many choices to make during the first few weeks living in the new place. Which of the questions is an example of an economic choice Paul might make? a. Where should I place my couch in the living room? b. Should I buy a new flat screen television set for my living room? c. Should I call my mom and talk with her about the problems I am having with my new bathroom? d. Should I invite my new neighbors over to share the home-warming cake my mom baked? 25. IS: Rules 26. Core Subject [Social Studies] Standard 3.9 -modifiedGiven an example of making an economic choice, determine three opportunity costs. 27. IS: Rules Valid Type(s) of Constructed Response Fill-in-the-Blank Labeling Short Answer Essay Project PerformanceValid Type(s) of Selected ResponseMultiple Choice Matching True/False Ordering 28. List three opportunity costs resulting from your decision to buy a new Nintendo Wii gaming system. 29. IS: Higher Order Rules 30. Core Subject [Social Studies] Standard 1.10The student will identify and apply the traits of good citizenship. 31. IS: Higher Order Rules Valid Type(s) of Constructed ResponseFill-in-the-Blank Labeling Short Answer Essay Project PerformanceValid Type(s) of Selected Response Multiple Choice Matching True/False Ordering 32. After watching a video clip from the movie Toy Story, describe three different ways in which Woody demonstrated good citizenship. 33. A 34. Core Subject [Social Studies] Standard 1.10 Value honesty and truthfulness in oneself and others. Core Subject [Social Studies] Standard 1.10 -modifiedThe student will choose to apply the traits of a good citizen. 35. Attitudes Valid Type(s) of Constructed Response Fill-in-the-Blank LabelingShort Answer Essay ProjectPerformanceValid Type(s) of Selected Response Multiple Choice Matching True/False Ordering 36. Teacher: I am going to put a star sticker next to your name every time I notice you acting like a Good Citizen in this class. 37. MS 38. Core Subject [Social Studies] Standard 1.5The student will construct a simple map of a familiar area, using basic map symbols in the map legend. Note: At one level, this is a higher-order rule. At a first-grade level, this also involves motor skills associated with drawing in general. 39. Motor Skills Valid Type(s) of Constructed Response Fill-in-the-Blank Labeling Short Answer Essay ProjectPerformanceValid Type(s) of Selected ResponseMultiple Choice Matching True/False Ordering 40. Using anything you want from the Arting table, make a map of your neighborhood. Dont forget to include basic mapping symbols, and include a legend. 41. Step ThreeFollow the rules 42. Rules based on Evidence of Validity: Ensure items address targeted Skills, Knowledge, and Attitudes/Dispositions (SKA) Maximize Grading efficiency (time and accuracy) Maximize probability that correct answers reflect SKA performance (minimize correct answer guesses - Type I Error False Positive) Maximize probability that incorrect answers reflect SKA performance (Type II Error False Negative) Encourage thinking about the questions 43. 1. General Rule 1.1: Write clear, concise, simple directions free of complex syntax or difficult vocabulary.Example: Bad Directions Newtons Three Laws of Motion defined an entire field of physical science for many years. Name them. Better Directions Name Newtons Three Laws of Motion. 44. 1. General Rule 1.2: All assessment items must clearly elicit the performances articulated within the objectives under the same conditions stated. Example: Objective: Given the mass of an object (in kilograms) and the acceleration of the object (in m/s2), the learners will calculate the force in Newtons. 45. 1. GeneralBad AssessmentSuppose a 1500 gram ball was thrown off the roof of a 2-story (approximately 7 meters high) building. What force will the ball apply to the ground when it hits (in Newtons)? Better Assessment Suppose a 1.5 Kg ball was thrown against a wall with an acceleration of 2 m/s/s. With what force will the ball hit the wall (in Newtons)? 46. 1. General Rule 1.3: Include reminders to perform certain steps if needed. Example: Draw a picture of a typical animal cell and label all the cell parts. Dont forget to draw very clear lines from your labels to the cell parts (use arrows if needed). 47. 1. General Rule 1.4: Included clear evaluation criteria if the assessment tasks are particularly complex. Example: Explain possible mechanisms involved in producing elevated global temperatures as a result of the greenhouse effect. Make certain that your answer references the chemical characteristics of hydrocarbons in the atmosphere, the flow of energy resulting in atmospheric temperature changes, and at least 3 specific natural and 3 specific artificial sources of hydrocarbon emissions contributing to changes in atmospheric concentrations of hydrocarbon gases. 48. 2. Multiple Choice Rule 2.1: Do not use any choices that are obviously incorrect. Example: How many centimeters are in a meter? a) .001 b) 1.00 c) 100 d) 3.12 49. 2. Multiple Choice Rule 2.2: Ensure that words such as a, an, he, she, or plural words are not used to cue learners toward the correct answer. Example: Pierre de Fermat is best known as a a) Inventor b) Scientist c) Mathematician d) Teacher 50. 2. Multiple Choice Rule 2.3: Dont use All of the Above.- If one choice is wrong, all of the above is wrong - None of the above is ok 51. 2. Multiple ChoiceRule 2.4: Dont use words like All or Never in the answer options.These terms almost always signify an answer option that is incorrect. 52. 2. Multiple Choice Rule 2.5: Dont repeat words from the question in the answer options.Example: Which of the following represents the best description of Natural Selection? a)The natural process responsible for the origin of new species and the adaptation of organisms to their environments. b)The procedures involved in cultivating the best possible type of offspring from a population of adult organisms. c)The adaptation of organisms their environment. 53. 2. Multiple Choice Rule 2.6: Dont use negative answer options following a negative question or stem.Example: Which of the assumptions about Natural Selection is NOT true? a)Organisms reproduce b)Traits vary among offspring c)Offspring do inherit traits from their parents d)The environment does not limit the size of populations. 54. 2. Multiple Choice Special Note: Paired MC items CAN be used to measure higher-order thinking skills:Example: 1. Suppose a frozen turkey was thrown from an airplane at 5,000 meters 2. Which of the following BEST explains why the answer chosen for item 1 is true? 55. 2. Multiple ChoiceSee Assessment Guide for More