ksta pre-conference november 5, 2009 welcome!. atlast item writing workshop horizon research, inc....
TRANSCRIPT
KSTA PRE-CONFERENCENOVEMBER 5, 2009
Welcome!
ATLAST ITEM WRITING WORKSHOP
Horizon Research, Inc.May 15-16, 2009
Made possible by a grant from the National Science Foundation (Grant no. EHR-0335328). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the presenters and do not necessarily reflect the views of the National Science Foundation.
Classroom Vignette
After a unit on seasons, a science teacher included the following item on a test:
Which of the following locations would you expect to have the least
temperature variation throughout the year?(a) Toronto, Canada(b) London, England(c) Sydney, Australia(d) Manta, Ecuador
The teacher had done a thorough job teaching the unit, and based on student responses to questions she asked during class, she was confident they would recognize a location on the equator as the correct answer. She was disappointed and confused when three-fourths of the class missed the item. As she returned the tests, she asked several students why they got the question wrong. Almost without exception, each student asked, “Where’s Ecuador?”
Excerpted from: Taylor, M. & Smith, S. (2009). How do you know if they’re getting it? Writing assessment items that reveal
student understanding. Science Scope, 32(5), 60–64.
Session Goals
1. Familiarity with a process for developing student assessment items.
2. Familiarity with some principles for developing student assessment items.
3. Opportunity to apply the process and principles for developing student assessment items.
Overview of Day
Overview an assessment development process
Clarifying the content domain Principles for writing student
assessment items Practice applying principles Write items in small groups Revise items in large group Orient to interviewing
A PROCESS FOR DEVELOPING ASSESSMENT ITEMS
Identify Science Topicsand MisconceptionsAbout Those Topics
Write MultipleChoice Items
InterviewStudents andRevise Items
Pilot Items
CreateAssessment
Activating Prior Knowledge
To write sound test items, what needs to be considered?
Think – Write – Pair – Share
Taylor, M. & Smith, S. (2009). How do you know if they’re getting it? Writing assessment items that reveal student understanding. Science Scope, 32(5), 60–64.
Use organizer to capture key ideas from the article.
Science Scope Article
CLARIFYING THE SCIENCE CONTENT TO BE ASSESSED
Assessment Development
Clarifying the Science Content
In ATLAST, we started with middle grades benchmarks.
Flow of matter and energy in living systems
Force and motion Plate tectonics
Overview of Benchmark Clarifications
Within each benchmark, we identified narrowly defined sub-ideas.
Sub-ideas are the smallest assessable piece of information
Within each sub-idea, we identified relevant common student misconceptions.
Why define the content domain?
It affects the validity of the assessment.
The content domain determines the boundaries of the assessable content.
CULLING STUDENT MISCONCEPTIONS RELATED TO THE SCIENCE CONTENT
Why identify common student misconceptions?
They are a source of plausible distractors for assessment items.
Sources of Student Misconceptions
Syntheses of researched misconceptions Content experts and educators with K-12
experience K-12 students Benchmarks for Science Literacy Atlas for Science Literacy Making Sense of Secondary Science
Eliciting Student Ideas
Answers to open-ended questions inform teachers of how students think about the content
Selection of misconception distractor inform teachers of how students think about the content
SOME PRINCIPLES FOR ITEM WRITING
Some Principles for Item Writing
Content Facilitating comprehension Structure Format Avoiding test-wiseness
CONTENT
Content
1. Target the item; one sub-idea per item if possible.
Why? Precision Good coverage of all sub-ideas within the
benchmark Cognitive load gets higher as more sub-ideas
are being assessed within one item
Content
2. Necessity principle: The knowledge in the sub-idea is needed to answer the item, so the right answer should be about the sub-idea.
The Necessity Principle
The knowledge in the sub-idea is needed to answer the item.
OR
The item cannot be answered correctly without knowing the sub-idea.
Practice Applying the Principle: Items 1A, 1B, and 1C
For each item,… Read the targeted sub-idea, correct
answer, and principle(s) being violated
Review the item
Identify/highlight why the item violates the principle(s)
Item 1A-Flow of Matter and Energy
Which one of the following energy transformations happens in a plant?
A. Light energy is transformed into chemical energy.
B. Heat energy is transformed into chemical energy.
C. Heat energy is transformed into motion energy.
D. Motion energy is transformed into heat energy.
Item 1B-Force and Motion
A student throws a ball straight up. Which one of the following best describes the motion of the ball as it falls down?
A. The ball moves faster and faster because of the force due to gravity.
B. The ball moves slower and slower because the force from the kick is acting against gravity.
C. The ball moves slower and slower because the force of motion is opposing the force due to gravity.
D. The ball moves at a constant speed because gravity is constant.
Item 1C-Plate Tectonics
In which one of the following places would it be most likely for an earthquake to happen?
A. Along the Pacific coast of North America
B. Along the Atlantic coast of North America
C. In the middle of the Pacific Ocean
D. In the middle of North America
Content
3. Sufficiency principle: The knowledge in the benchmark is ALL a student needs to know to answer the item correctly.
The distractors may be about other sub-ideas as long as they’re within the benchmark, but it’s best to have distractors also be in the sub-idea.
The Sufficiency Principle
The knowledge in the benchmark is ALL a student needs to know to answer the item correctly.
OR
Answering the item correctly requires NO knowledge outside the benchmark.
Force and Motion Benchmark
An unbalanced force acting on an object changes its speed or direction of motion or both.
Example:
Which of the following is a force that opposes motion?
A. AccelerationB. InertiaC. MomentumD. Friction
Practice Applying the Principle: Items 2A, 2B, and 2C
For each item,… Read the targeted sub-idea, correct
answer, and principle(s) being violated
Review the item
Identify/highlight why the item violates the principle(s)
Item 2A-Flow of Matter and Energy
Which one of the following is NOT food for humans?
A. SugarsB. FatsC. ProteinsD. Vitamins
Item 2B-Force and Motion
A girl slides a box across the floor. Which of the following is a force that acts on the box?
A. The energy the girl transfers to the box
B. The momentum of the box
C. The inertia of the box
D. The friction between the floor and the box
Item 2C-Plate Tectonics
Why do earthquakes occur more often in some areas than in others?
A. Earthquakes tend to occur where there are large oil deposits underground.
B. Earthquakes tend to occur in desert regions, where the ground is unstable.
C. Earthquakes tend to occur near the edges of plates.
D. Earthquake location is currently not well understood by scientists.
Content
4. Item should not teach content.
Example:
A girl gets on her skateboard and pushes off with her foot. As she moves across the gym floor, she moves slower and slower and eventually comes to a stop. Why does she come to a stop?
A. Because the force due to her foot runs out.B. Because the force in the skateboard runs out.C. Because the force of friction is acting on the skateboard.D. Because the force due to gravity is greater than the force due to friction.
Correction:
A girl gets on her skateboard and pushes off with her foot. As she moves across the gym floor, she moves slower and slower and eventually comes to a stop. Why does she come to a stop?
A. Because the force due to her foot runs out.B. Because the force in the skateboard runs out.C. Because the force of friction is acting on the skateboard.D. Because there are no forces acting on the skateboard.
Practice Applying the Principle: Items 3A, 3B, and 3C
For each item,… Read the targeted sub-idea, correct
answer, and principle(s) being violated
Review the item
Identify/highlight why the item violates the principle(s)
Item 3A-Flow of Matter and Energy
Which one of the following do plants NOT need to make food?
A. WaterB. LightC. Carbon dioxideD. Soil
Item 3B-Force and Motion
A boy slides a salt shaker along a table toward the right. Friction acts to oppose the motion of the salt shaker. As the salt shaker slides, in which direction does the force of friction act on the salt shaker?
A. To the left
B. To the right
C. Upward
D. Downward
Item 3C-Plate Tectonics
Which one of the following best describes how fast Earth’s plates move?
A. A few centimeters a yearB. A few centimeters a monthC. A few meters a yearD. A few meters a month
Content
5. Mirroring: Answer choices should reflect the question being asked.
Example:
How do plants get their food?
A. Plants make food from minerals, water, and sunlight.
B. Plants make food from carbon dioxide, water, and sunlight.
C. Plants get food by absorbing water through their leaves.
D. Plants get food by absorbing nutrients through their roots.
Example:
A sailboat is traveling in the ocean, as shown in the diagram. The wind exerts a force that pushes the boat forward (toward the right side of the page), and the water exerts a weaker force to the left.
How do these forces interact?
A. The forces combine to create a total forward force that is smaller than the force due to the wind.B. The forces combine to make a total forward force that is larger than the force due to the wind.C. The forces cancel each other out.D. The force to the left doesn't matter because the force to the right is stronger.
AVOIDING TEST-WISENESS
Avoiding Test-wiseness
Students tend to…
Choose the longest answer
Choose the most “science-y” answer
Rule out answers with strong qualifiers (e.g., only, never, always)
Practice Applying the Principle: Items 4A, 4B, and 4C
For each item,… Read the targeted sub-idea, correct
answer, and principle(s) being violated
Review the item
Identify/highlight why the item violates the principle(s)
Item 4A-Flow of Matter and Energy
Grass is food for cows. How does the cow use the grass?
A. All the grass leaves the cow’s body after the energy is absorbed.B. All the grass leaves the cow’s body after the water is absorbed.C. The grass is broken down and some of it becomes part of the cow’s body.D. The grass is broken down, but none of it becomes part of the cow’s body.
Item 4B-Force and Motion
A school bus is slowing down as it comes to a stop sign. What is true about the forces acting on it while it is slowing?
A. The forces in the forward direction are greater than the forces in the backward direction.B. The forces in the forward direction are less than the forces in the backward direction.C. The only force acting on the bus is friction.D. There are no forces acting on the bus.
Item 4C-Plate Tectonics
Which one of the following is true about Earth’s plates?
A. Earth has one large plate of solid rock.B. Earth has one large plate that has some solid parts and some liquid parts.C. Earth has a number of plates of mostly solid rock.D. Earth has a number of plates, all of which are liquid.
FACILITATING COMPREHENSION
Facilitating Comprehension
7. Keep sentences brief and straightforward, with a simple phrase structure and no additional clauses.
Example:
The South American plate and the Nazca plate are neighboring plates, with no other plates in between. Scientists say that the South American plate and the Nazca plate are moving toward one another.
Which one of the following best explains what they mean when they say that these two plates are moving toward one another?
Facilitating Comprehension
12. Use present tense and active voice only.
Example:
A shopping cart is pushed so that it moves faster and faster. What is true about the situation?
Correction:
A man is pushing a shopping cart. The cart is moving faster and faster. What must be true about the situation?
Facilitating Comprehension
12. Visuals should facilitate the understanding of what is being asked; don’t include a visual just to include a visual.
No extraneous information Should complement the presented scenario Labels should mirror wording in the item
STRUCTURE
Structure
Restricted to four answer choices
Avoid multiple correct answers (i.e., do not use “Both A and B are correct”)
Avoid using “None of the above”
FORMAT
Practice Applying the Principle: Items 5A, 5B, and 5C
For each item,… Read the targeted sub-idea, correct
answer, and principle(s) being violated
Review the item
Identify/highlight why the item violates the principle(s)
Item 5A-Flow of Matter and Energy
When a baby chick develops inside an egg, the yolk in the egg is its only source of food. As the chick grows, the yolk decreases in size. Why does the yolk decrease in size?
A. Matter is destroyed during the growth of the chick. B. The yolk provides the building blocks the chick is made of. C. The yolk is a source of energy for the chick but not a source of body parts. D. The yolk decreases in size to make room for the growing chick.
Item 5A-Flow of Matter and Energy
When a baby chick develops inside an egg, the yolk in the egg is its only source of food. As the chick grows, the yolk decreases in size. Why does the yolk decrease in size?
A. Matter is destroyed during the growth of the chick. B. The yolk provides the building blocks the chick is made of. C. The yolk is a source of energy for the chick but not a source of body parts. D. The yolk decreases in size to make room for the growing chick.
Item 5B-Force and Motion
A car is moving faster and faster along a straight, level section of road. Which of the following must be true about the forces on the car?
A. The total force in the direction of the car’s motion is greater than the total force in the opposite direction.B. There is an increasing total force in the direction of the car’s motion.C. The total force in the direction of the car’s motion is greater than the car’s force.D. There are no forces in the direction opposite to the car’s motion.
Item 5B-Force and Motion
A car is moving faster and faster along a straight, level section of road. Which of the following must be true about the forces on the car?
A. The total force in the direction of the car’s motion is greater than the total force in the opposite direction.B. There is an increasing total force in the direction of the car’s motion.C. The total force in the direction of the car’s motion is greater than the car’s force.D. There are no forces in the direction opposite to the car’s motion.
Item 5C-Plate Tectonics
Locations 1, 2, and 3 are shown on a map of Iceland to the left. The arrows indicate directions of plate movement.
Where would you expect to find the youngest rock?
A. Location 1B. Location 2C. Location 3D. The rock is the same age at all 3 sites.
Item 5C-Plate Tectonics
Locations 1, 2, and 3 are shown on a map to the left. The arrows indicate directions of plate movement.
Where would you expect to find the youngest rock?
A. Location 1B. Location 2C. Location 3D. The rock is the same age at all 3 sites.
Most Important Rules
1. Target the item; one sub-idea per item if possible.
2. Necessity principle: The knowledge in the sub-idea is needed to answer the item, so the right answer should be about the sub-idea.
3. Sufficiency principle: The knowledge in the benchmark is ALL a student needs to know to answer the item correctly. The distractors may be about other sub-ideas as long as they’re within the benchmark, but it’s best to have distractors also be in the sub-idea.
Content-Specific Guidelines
Developed during the process of item writing to address issues that arose in student and/or content expert feedback
Please review prior to writing items
WRITING ITEMS
Assessment Development
Writing Items
Goals and Guidelines Helpful hints Logistics
Goals and Guidelines
Goal: Write 2-3 items per pair/group If you want to work independently for a
portion of the time, please make sure you move toward collaboratively drafting, reacting to, and/or editing items after the first 20 minutes.
Use the item writing checklist to “interview” your own items
“Interview” Your Items
Have you clearly defined the content you are assessing? Do you have a good sense of what is fair game?
Does the item clearly target a specific idea? Do students have to know the content to answer
the item? Could they answer it correctly without knowing the content?
Is the target content sufficient for students to be able to answer the item? Do they need to know anything outside the target content?
“Interview” Your Items
Does the item inform students about the correct answer to other items?
Do the answer choices mirror the wording of the item, or is there a mismatch?
Does the item: keep sentences as brief as possible? use present tense and active voice only? make appropriate use of visuals?
Can students answer the item without knowing the content by using common test-taking strategies?
Helpful Hints
Use existing items as fodder for new items (your own or samples).
Use the “related prior student conceptions” as a source of distractors.
Use textbooks, diagrams, etc. to get ideas for items around specific sub-ideas.
GROUP EDITING OF ITEMS
Attack the item; affirm the writer.
OVERVIEW OF STUDENT COGNITIVE INTERVIEW PROCESS
Assessment Development
Purposes
Cognitive interviews help…
1. Determine the item’s validity2. Determine the item’s clarity3. Determine the item’s plausibility; and4. Identify specific improvements/
revisions to the item
Purposes
1. Validity Does the student think the item is asking
what we intended it to ask? Does the student use the knowledge we
are trying to measure to answer the item? Is the student getting the item
correct/incorrect for the appropriate reason (i.e., understanding/not understanding the knowledge being measured rather than the wording being tricky or because of test wise-ness).
Purposes
2. Clarity• To make sure the item is clear and the
wording and/or graphics are not confusing
3. Plausibility• To determine if the scenario is realistic
4. Improvements• To elicit suggestions for alternative
distractors (e.g., one that may be more realistic or better aligned with a common misconception)
Interview Protocol
Prologue
Purpose of the prologue Helps explain what we are doing; Helps put the student at ease:
Interview is voluntary; Interview is anonymous; Stresses that we aren’t testing him/her, rather
we’re trying to determine if the items are any good.
Background Information
Helps put responses into context; Allows us to internally track:
who has been interviewed; around which items; and when.
Procedure
Ask student to read and think aloud as they read the item and consider the answer choices. Gives you a sense if wording is tricky (e.g.,
s/he re-reads the item several times); and Helps you determine why student is
answering the way s/he is (e.g., real understanding, guessing, test wise-ness).
INTERVIEW: PROCEDURE
Item-Specific Questions
1. Why did you choose that answer? [Validity]
2. What did you think of each of the other answer choices? [Validity]
3. Was there an answer choice you were expecting to see, but did not? [Improvements]
Item-Specific Questions
4. Were there any words or diagrams you did not really understand, or situations that made the question confusing? [Clarity and Plausibility]
5. Is there anything about the question that did not confuse you, but that you think might confuse other middle school students? [Clarity and Plausibility]
INTERVIEW NOTES
Interview Notes
Place to capture student thinking Record of which answer choices were
selected, considered, and/or ruled out Coding of item—Did the student answer
the question correctly or incorrectly for the right or wrong reason? (record after the interview)
STUDENT INTERVIEW
INTERVIEW: FORCE AND MOTION ITEM
Force and Motion Item
Sub-idea E: If an object is slowing down, then there is a net force acting on the object in the direction opposite to the object’s motion.
Force and Motion Item
Tim stretches a rubber band between two posts and uses it to launch a toy car, as seen in the picture. He pulls the car back against the rubber band (as shown in 1), then lets go and the car moves across the floor (as shown in 2 and 3). The car slows down and eventually stops after a few meters.
Why does the car slow down and stop?
Force and Motion Item
Why does the car slow down and stop?
A. The force due to the rubber band gets used up.
B. The car’s force of motion eventually runs out.
C. There is a force acting in the direction opposite to the car’s motion.
D. The car starts to slow down once the backward forces are equal to the forward forces.
Force and Motion Item
The student chose the incorrect answer for the right reason—she did not fully understand the content
Content probing re: friction
Asked about all answer choices
PREPARING FOR STUDENT COGNITIVE INTERVIEWS
Interview Tips
Reassure/reinforce good responding, especially early in interview.
Don’t let the interviewee know what the correct answers are during the interview; you can review items after the interview if student desires.
Remember the main goal of interview is to check validity of item, not to uncover student thinking about the content.
Interview Tips
Are there any answer choices that you can definitely rule out?
Are there any new, unfamiliar, or confusing words in the question or answer choices?
Let’s just talk about each answer choice and what you thought they meant by it.
Let’s say you were taking one of those standardized tests and you just had to pick one answer to bubble in on the answer sheet. Which one would you choose?
I’m curious about what they meant by “X.” What do you think they meant by “X”?
I noticed they used the word “X.” If you had to explain to a younger student what “X” meant, what would you say?
TAKE HOME MESSAGES ON ITEM WRITING
Item Writing
There are many item writing principles; a few are key to writing valid assessments.
1. Clarify the content domain2. Target item3. Necessity 4. Sufficiency
Next Steps in the Assessment Development Process
Identify Science Topicsand MisconceptionsAbout Those Topics
Write MultipleChoice Items
InterviewStudents andRevise Items
Pilot Items
CreateAssessment
Reflection
On an index card, write 3 things you learned today 2 things you plan to try 1 question you still have