Slide 1

Developing High Quality Items and AssessmentsOn PURPOSE!Keystone Item and Test DevelopmentIU1 November 27, 2012Keystone blueprints & Keystone resultsWebbs Depth of KnowledgeMultiple-choice item writing rulesPractice writing multiple-choice itemsLUNCHConstructed-response item writing rulesPractice writing constructed-response items

AGENDAWhy?Understand the quality and rigor of the Keystone exams

Many districts are developing their own Keystone-like items and assessmentsWhats the purpose of the test? What are you going to do with the results?What decision(s) will be made based on the results?Prepare students for Keystone-like items.

Whats the content of the test?Algebra I Assessment AnchorsBiology Assessment AnchorsEnglish Literature Assessment AnchorsAt a higher cognitive complexity (Depth of Knowledge)Two Important Questions Purpose and ContentDefining the PurposeA test is only reliable and valid for the purpose for which it was designed.Shula Nedley, Ph.D.

The items must be comparable to the items on the Keystones.

Therefore, we need to write items to match the Keystone specifications.

Each Keystone has two Modules

Each Keystone has both multiple-choice and constructed-response items.Introducing.the Test BlueprintModule 1Module 2Assessment Anchors CoveredOperations and Linear Equations and InequalitiesLinear Functions and Data Organizations# of Eligible Content Covered1815# of Multiple Choice Questions1818# of Constructed Response Questions33# of Multiple Choice Points1818# of Constructed Response Points1212Algebra I Keystone from PDE documents - part of the Blueprint60%40%Module 1Module 2Assessment Anchors CoveredCells and Cell ProcessesContinuity and Unity of Life# of Eligible Content Covered1622# of Multiple Choice Questions2424# of Constructed Response Questions33# of Multiple Choice Points2424# of Constructed Response Points99Biology Keystone from PDE documents - part of the Blueprint73%27%Module 1Module 2Assessment Anchors CoveredFiction LiteratureNonfiction Literature# of Eligible Content Covered2531# of Passages22# of Multiple Choice Questions1717# of Constructed Response Questions33# of Multiple Choice Points1717# of Constructed Response Points99Literature Keystone from PDE documents - part of the Blueprint65%35%Spring 2011 Statewide Results38.6%Webbs Depth of KnowledgeWebb & BloomDoK vs DifficultyWebbs Depth of KnowledgeWebb developed a process and criteria for systematically analyzing the alignment between standards and assessments.

Webbs Depth of Knowledge Model can be used to analyze the cognitive complexity required for the student to master a standard or complete an assessment task.

12Webbs Depth of KnowledgeCognitive complexity refers to the cognitive demand associated with a test question.

The Depth of Knowledge level of the question is determined by the complexity of the mental processing that the student must use to answer the question.

131. Recall - Recall of a fact, information, or procedure2. Basic Application of Skill/Concept - Use of information, conceptual knowledge, procedures, two or more steps, etc.3. Strategic Thinking - Requires reasoning, developing a plan or sequence of steps; has some complexity; more than one possible answer; generally takes less than 10 minutes to do.4. Extended Thinking - Requires an investigation; time to think and process multiple conditions of the problem or task; and more than 10 minutes to do non-routine manipulations.

Webbs four levels of complexity14Handout on Norm Webbs Depths of KnowledgeSometimes verbs help us write items at different DOK levels

Cognitive Level Comparison Matrix: Bloom And WebbAction WordsBLOOMWEBB1.0define, identify, name, select, state, order (involves a one-step process)1.0define, identify, name, select, state, order (involves a one-step process)2.0convert, estimate, explain, express, factor, generalize, give example, identify, indicate, locate, picture graphically (involves a two-step process)2.0apply, choose, compute, employ, interpret, graph, modify, operate, plot, practice, solve, use (involves a two-step process)3.0apply, choose, compute, employ, interpret, graph, modify, operate, plot, practice, solve, use, (involves a three-or-more step process)4.0compare, contrast, correlate, differentiate, discriminate, examine, infer, maximize, minimize, prioritize, subdivide, test3.0compare, contrast, correlate, differentiate, discriminate, examine, infer, maximize, minimize, prioritize, subdivide, test5.0arrange, collect, construct, design, develop, formulate, organize, set up, prepare, plan, propose, create, experiment and record data4.0arrange, collect, construct, design, develop, formulate, organize, set up, prepare, plan, propose, create, experiment and record data6.0appraise, assess, defend, estimate, evaluate, judge, predict, rate, validate, verifyLevel 2 items are measuring more than one skillWhat are the skills this item is measuring?Level 3 items go even deeper and measure more than two skills. Multiple skills must be known to correctly answer the item.In PA Level 3 DOK are measured primarily with Constructed R items.In PA Level 4 DOK are long term projects.But.dont get stuck on the verbsSame VerbThree Different DOK Levels

DOK 1- Describe three characteristics of metamorphic rocks. (Requires simple recall)

DOK 2- Describe the difference between metamorphic and igneous rocks. (Requires cognitive processing to determine the differences in the two rock types)

DOK 3- Describe a model that you might use to represent the relationships that exist within the rock cycle. (Requires deep understanding of rock cycle and a determination of how best to represent it)

The word illeist means a person who:

refers to themselves in the 3rd person.plays a specialized musical instrument.is sick.loves animals.DOK DifficultyDifficulty: Easy, Medium, Hard?DOK: 1, 2, 3?In Mr. Bells classes, the students voted for their favorite shape for a symbol. Here are the results.

Using the information in the chart, Mr. Bell must select one of the shapes to be the symbol. Which one should he select and why?The shape Mr. Bell should select: _________________Explain: _________________________________________DOK DifficultyClass 1Class 2Class 3Shape N91418Shape P1919Shape Q22142Difficulty: Easy, Medium, Hard?DOK: 1, 2, 3?By inserting a gene into crop plants, scientists have developed plants that are resistant to insects. If an insect eats the plant, the insect dies. Which practice is unnecessary with this new plant variety?

Eroding the land by tillingOverproducing food cropsRemoving weeds from cropsSpraying plants with pesticidesDOK DifficultyDifficulty: Easy, Medium, Hard?DOK: 1, 2, 3?You are trapped in a room with 2 doors, one of which leads to freedom and the other to certain death. You do not know which is which. Each door is guarded by a man, one of whom always lies, while the other always tells the truth. You do not know which is which. You can ask but one question of either man to gain your freedom. Which question should you ask?

How do I get out of this room and stay alive?Which one of the two door is the correct one?Who is the man who never tells the truth & always lies?Which door would the other man say leads to death?Test Your Depth of KnowledgeIdentifying good itemsBest practices for writing/reviewing multiple-choice itemsThe Test of FranzipanicsWho speaks Franzipanics?The Test of FranzipanicsTake 5 minutes and try to answer each question, individually.

Then, take another 5 minutes and share your answers at your table.Good Item Writing RulesWhat did we learn from Franzipanics?1. The purpose of the cluss in furmpaling is to removea. cluss-pragsb. tremalisc. cloughsd. plumots

Rule - Structure: Do NOT use a cue word in the stem and the keyed (correct) answer.REFER TO THE GOOD ITEM WRITING RULES

Cluss is the cue word.Florida Style Guide required: (1) Item stems to be questions (closed stems) and (2) Distractors to be Capital Letters followed by a period.A. B. C. D.26Good Item Writing RulesWhat did we learn from Franzipanics?2. Trassig is true when a. lusp trasses the vomb. the viskal flans, if the viskal is donwil or zortil c. the belgo frullsd. dissles lisk easily

Rule - Structure: Make distractors of equal length.Good Item Writing RulesWhat did we learn from Franzipanics?3. The sigla frequently overfesks the trelsum becausea. all siglas are melliousb. siglas are always votialc. the trelsum is usually tariousd. no trelsa are feskable

Rule - Structure: Do NOT use absolutes such as all, always, no. Stick with usually, frequently.Good Item Writing RulesWhat did we learn from Franzipanics?4. The fribbled breg will minter best with ana. derst b. morst c. sorterd. ignu

Rule - Structure: No grammatical hints such as an.

Good Item Writing RulesWhat did we learn from Franzipanics?5. Among the reasons for tristal doss area. the sabs foped and the foths tinzed b. the kredges roted with the orotsc. few rakobs were accepted in sluthd. most of the polats were thonced

Rule - Structure: No grammatical hints such as are or is which suggests plural or singular.

Good Item Writing RulesWhat did we learn from Franzipanics?6. Which of the following (is, are) always present when trossels are being gruven?a. rint and vostb. sot and plonec. shum and vostd. vost

Rule - Structure: No hints in the distractors. Repeating one word throughout the distractors provides a hint.

Good Item Writing RulesWhat did we learn from Franzipanics?7. The mintering function of the ignu is most effectively carried out in connection witha. raxma tol b. the groshing stantol c. the fribbled bregd. a frally sush

Rule - Structure: Students opportunity to correctly answer an item must be independent of their performance on another item. Do NOT link items.

Good Item Writing RulesWhat did we learn from Franzipanics?8. a. b. c. d.

Rule - Structure: Do NOT create patterns in your answer key.

with poorly written items, you can get them correct without knowing the intended content.

Reliability issue?

Validity issue?The moral of this testBREAK TIME!9:50 10:00Why are we writing items?To gauge student learning.To receive Act 48 hours.To work with other adults.To avoid my students.The Anatomy of the Multiple Choice Item

The StemDistractorDistractorDistractorCORRECT ANSWERMultiple-choice ItemsThe advantage is that, with careful construction, this type can be used to measure knowledge at most levels.

The disadvantage is that it's hard to write good distractors for levels beyond factual recallMultiple Choice Items

Multiple-Choice Item Writing GuidelinesRules regarding Structure: Multiple choice items will have four response options and only one correct answer.Response options should be parallel in reference to parts of speech.Response options should not overlap.

Response options should be parallel in reference to parts of speech. EXAMPLE

Why did the chicken cross the road?

To get to the other side.Walking then running through traffic to catch the bus.To demonstrate proficiency in barnyard standard 3.1.Chicken feed on the other side.

Response options should not overlap.

EXAMPLE

The average combined score in the NFL playoff games in 2010 was:50

Less than 10About 20About 40More than 50The problem with "A and B" as a response option is that it becomes more of a logic test than a knowledge or skills test.

42Multiple-Choice Item Writing GuidelinesRules regarding Structure: Avoid absolutes such as only or never.If the distractors are numeric, put them in either ascending or descending order. If the distractors are alphabetic and relatively short (one to three words), put them in alphabetical order.Response options should not include: None of the above, All of the above, not enough information, Cannot be determined.

Multiple-Choice Item Writing GuidelinesRules regarding Structure: Dont repeat the same word(s) at the beginning of each response add the word(s) to the stem. The item stem should be a complete thought. The easiest way to tell if the stem is a complete thought is to cover up the response options and see if you know what youre supposed to do. Students should not have to figure out what youre asking.

Is this item stem clear?Multiple-Choice Item Writing GuidelinesRules regarding Content: Each multiple-choice item should be written to measure only one eligible content statement.Items should be clear and concise, and they should use vocabulary and sentence structure appropriate for the assessed grade level.Distractors should be incorrect but plausible based on the topic of the question.Ensure that there is only one true and defensible answer.

Is there one defensible answer?

Multiple-Choice Item Writing GuidelinesRules regarding Content: Avoid jargon and textbook language.

Avoid clichs.

Use common misinformation purposefully.

Use logical misinterpretations purposefully.

Multiple-Choice Item Writing GuidelinesGet your thoughts down on paper as soon as possible. Then edit.Ensure that there is only one true and defensible answer.Have someone else review your work.

Some of us are good item writers

Some of us are good item reviewers and editorsWe all have our unique skills and talentsLets practice using the Item Writing Guidelines by reviewing and editing some items together.1. Multiple-choice items

may have several correct answers. consists of a stem and some options. always measure factual details. Whats the issue?How could the item be modified?stem and several distractors. correct answer and several distractors. stem, a correct answer, and some distractors. stem and a correct answer. 1. The components of a multiple-choice item are:2. The most serious aspect of the energy crisis is the possible lack of fuel for industry. possibility of widespread unemployment. threat to our environment from pollution. possible increase in inflation. cost of developing alternate sources of energy.Whats the issue?How could the item be modified?The item below would be a good basis for discussion but probably should not be included in an examination. Such an item might be rewritten to focus on a more specific and/or a particular aspect of the energy crisis. It might also be written to focus on the opinion of a recognized expert.

542. According to the National Energy Council, the most serious aspect of the energy crisis is the possible lack of fuel for industry. possibility of widespread unemployment. threat to our environment from pollution. possible increase in inflation. cost of developing alternate sources of energy.The item below would be a good basis for discussion but probably should not be included in an examination. Such an item might be rewritten to focus on a more specific and/or a particular aspect of the energy crisis. It might also be written to focus on the opinion of a recognized expert.

553. If a test is extremely easy for a given group, we would expect:The reliability to be moderate.The reliability to be high.The reliability to be unaffected by the student scores.The reliability to be low.

Whats the issue?How could the item be modified?3. If a test is extremely easy for a given group, we would expect the reliability to be:high.low.moderate.unaffected.

4. What is an acceptable reliability coefficient for teacher made classroom assessments?0.300.700.450.97

Whats the issue?How could the item be modified?4. What is an acceptable reliability coefficient for teacher made classroom assessments?0.300.450.700.97

As we write items, they will be identified in terms of their possible placement on the local aligned assessment.

Introducingthe Item CardIntroducingthe Item CardAssessment Anchor/Eligible Content Keystone Item CardItem ID (initials001)Content AreaScenario ID or Passage IDScenario Title or Passage TitleItem TypeDepth of KnowledgeEstimated DifficultySource of ChallengeRUBRIC (if applicable)GraphicsLanguage DemandBiasAnswer Key# of PointsDistractorsSource: Now its your turn by content areaEach content area has four (4) pre-selected eligible content statements, two (2) from Module 1 and two (2) from Module 2. In pairs, Select either the Module 1 or the Module 2 eligible content statements.Write two DOK level 1 items, one for each EC. Take what youve written and change the item to make it a level 2 DOK item.Trade your 4 items (2 DOK 1 and 2 DOK 2) with another pair in your content area.Constructively improve each others items.Content AnchorItem Writing Level 1Item Writing Level 21L.N.1.1.4Explain how an authors use of key words or phrases in text informs and inuences the reader.2L.N.2.1.2Cite evidence from a text to support generalizations.Item Level 1 CritiqueItem Level 2 Critique12Lunch Time!Coming up after lunch: Constructed Response ItemsThe question or the taskThe rubricPDE has defined general rubrics for Algebra I, Biology, and LiteratureBut each CR will also need a specific rubricAnnotated student work at each rubric score point is a big plus!Two parts of a constructed-response itemBe clear as to the content you expect the student to exhibit (rubric).Allow students to demonstrate their knowledge in new situations.Ask questions that are specific and focused.Word questions so that all students will interpret the task the way you intend. (Literature quiz)Word questions so that experts can agree on the correctness of a response.What can we learn from the PDE released items?General Principles for writing CR itemsConstructed Response ItemsAlgebra IBiologyLiteratureCR items written to the Assessment AnchorsCR items written to the Eligible ContentCR items written to the Eligible Content10 minutes8 minutes5 minutes + time to read passageRubric: 0 to 44 is Correct ResponseRubric: 0 to 33 is Correct ResponseRubric: 0 to 33 is Correct ResponseUsually 3 to 4 parts with 1 to 2 points per part Usually 3 parts with 1 point per partOne part worth up to 3 pointsScaffolded CR parts build on each other Extended CR - require students to explain, WHY?

Develop a solution to a biology problemUse information/examples from text to support your answer/position/analysis.DOK 3DOK 3DOK 3Scaffolded CR - Algebra IA1.2.3The weight (in pounds) of each wrestler on the high school wrestling team at the beginning of the season is listed below.178 142 112 150 206 130What is the median weight of the wrestlers? (1 point)What is the mean weight of the wrestlers? (1 point)Two more wrestlers join the team during the season. The addition of these wrestlers has no effect on the mean weight of the wrestlers, but the median weight of the wrestlers increases 3 pounds. Determine the weight of the two new wrestlers. (2 points)Biology CR sample itemBIO.A.1.2.1Prokaryotic cells are generally much smaller than eukaryotic cells.Identify a structural difference between prokaryotic cells and eukaryotic cells that is directly related to their difference in size. (1 point)Based on the structural difference, explain why prokaryotic cells can be much smaller than eukaryotic cells. (1 point)Describe one similarity between prokaryotic cells and eukaryotic cells that is independent of size. (1 point)Literature CR sample itemL.F.2.3.1Analyze what Howards thoughts throughout the passage reveal about his personality. Use information from the passage to support your analysis. (3 points)

L.N.2.5.6Analyze how the author constructs the argument in the passage. Use information from the passage to support your analysis. (3 points)Now its your turn by content areaWe want you to write ONE constructed response item.For Biology and Literature, we selected one eligible content statement that you wrote multiple-choice items to this morning. In triads, Write one DOK level 3 item. Write the item and the accompanying rubric/scoring information.Trade your item with another triad in your content area.Constructively improve each others item.Post your best CR item for a gallery walk.Now its your turn by content areaFOR ALGEBRA I: We selected the Assessment Anchor that you are going to work with. That anchor includes one of the eligible content statements that you wrote MC items to this morning.In triads, Write one DOK level 3 item. Write the item and the accompanying rubric/scoring information.Trade your item with another triad in your content area.Constructively improve each others item.Post your best CR item for a gallery walk.

Look at the Assessment Anchors and Eligible Content with sample items.Look at the Assessment Anchor Item Samplers for prototype itemshttp://www.pdesas.org/ Keystone Exams Item and Scoring SamplersAre your items of similar difficulty?Are your items of similar DOK?Before beginning to write itemsGoogle search:[content area] Released Items or Released Test QuestionsPublic Domain [texts]Common Core Appendix B for public domain passages by grade level.

Dont assume that items are appropriate for the Keystones. Review against the guidelines and against the Eligible Content.

Sources for itemsQUESTIONS or for copies of the PowerpointJim Turner, OnHand Schoolsjwt7@pitt.edu

Shula Nedleydrshula@comcast.net