writing constructed response assessment items:

Writing Constructed Response Assessment

Items:

One Element of a Comprehensive Assessment System

What are Constructed Response Items?

• Open-ended Questions

• Require several sentences or brief paragraph

• Require higher level thinking (than simple recall) and the application of students’ knowledge Making Comparisons Identifying Patterns Evaluating Points of View Making Generalizations Synthesizing Information

• Allow for the examination of Student Thinking

• Scored using a Rubric that provides varying degrees of Credit

Wisconsin Dept. of Public Instruction, May 2007

Why Use Constructed Response Items?

“The notion that learning comes about by accretion of little bits is outmoded learning theory. Current models of learning … contend that learners gain understanding when they construct their own knowledge and develop … interconnections among facts and concepts …”

(Shepard, 1989. quoted in Michigan Curriculum Framework)

Constructed Response as part of a Coordinated Assessment

System:

“An assessment system is an ongoing coordinated process of collecting information for the purpose of continuously improving student learning.”

Michigan Curriculum Framework

Elements of a Coordinated Assessment System:

Fixed Response:

Simple items that assess factual information / discrete skills. Typically there is a single best answer.

Constructed Response:

Short answers, explanations, essays or diagrams that involve analysis or evaluation. Require judgment-based scoring (i.e. rubrics).

Oral Questioning in Class: Could be fixed or constructed response.

Teacher Observation: Typically involve a simple checklist.

Performance: Speeches, experiments, debates, etc. Require judgment-based scoring.

Project: Designing and/or building useful things. May be done individually or collectively.

Portfolio: A collection of student work over time – written work, artistic creations, project reports, etc.

Standardized Monitoring: e.g. MEAP/MME, ACT, SAT, NWEA, etc.

Adapted from Michigan Department of Education, Science Education Guidebook; and Wiggins, G. & McTighe, J (1998). Understanding by Design. Association for Supervision and Curriculum

Development (ASCD).

WHEN to use Constructed Response:Consider the VERB of the Content Expectation:

• B2.5i Relate cell parts/organelles to their function.

• P2.1A Calculate the average speed of an object using the change of position and elapsed time.

• C5.5A Predict if the bonding between two atoms of different elements will be primarily ionic or covalent.

• E5.3e Determine the approximate age of a sample, when given the half-life of a radioactive substance along with the ratio of daughter to parent substances present in the sample.

Can be assessed with FIXED RESPONSE Items


• B1.1A Generate new questions that can be investigated in the laboratory or field.

• P1.1g Use empirical evidence to explain and critique the reasoning used to draw a scientific conclusion or explanation.

• C5.8B Draw isomers for simple hydrocarbons.

• E5.4C Analyze the empirical relationship between the emissions of carbon dioxide, atmospheric carbon dioxide levels and the average global temperature over the past 150 years.

Best assessed with CONSTRUCTED RESPONSE Items


• B1.1C Conduct scientific investigations using appropriate tools and techniques.

• P1.2C Develop an understanding of a scientific concept by accessing information from multiple sources. Evaluate the scientific accuracy and significance of the information.

Best assessed with a PERFORMANCE exercise

• E1.2D Evaluate scientific explanations in a peer review process or discussion format.

Best assessed through TEACHER OBSERVATION

HOW to use Constructed Response Items:

Standards for Assessment Tasks:1. Organization of Information —The task asks students to organize,

synthesize, interpret, explain, or evaluate complex information in addressing a concept, problem or issue.

2. Consideration of Alternatives —The task asks students to consider alternative solutions, strategies, perspectives, or points of view in addressing a concept, problem or issue.

3. Disciplinary Content —The task asks students to show understanding and/or use ideas, theories, or perspectives considered central to an academic or professional discipline.

4. Disciplinary Process —The task asks students to use methods of inquiry, research, or communication characteristic of an academic or professional discipline.

5. Elaborated Written Communication —The task asks students to elaborate on their understanding, explanations, or conclusions through extended writing.

6. Problem Connected to the World Beyond the Classroom —The task asks students to address a concept, problem or issue that is similar to the one that they have encountered or are likely to encounter in life beyond the classroom.

7. Audience Beyond the School —The task asks students to communicate their knowledge, present a product or performance, or take some action for an audience beyond the teacher, classroom, and school building.

Michigan Curriculum Framework

HOW to Develop Constructed Response Items:

The Specifics• Set the Context

Specify the knowledge to be brought to bear

• Specify the Reasoning Use specific verbs e.g. analyze, cite, describe…

• Point the Way Inform students of the criteria that will be applied to

evaluate their responses

• Develop the Scoring Rubric Clear articulation of the appropriate evaluation criteria

by which to judge the quality of student responses.

Adapted from Classroom Assessment for Student Learning, Richard Stiggins, et. al. 2006

Examples: 5th Grade Science

E.ST.05.22 Explain moon phases as they relate to the position of the moon in its orbit around the Earth, resulting in the amount of observable reflected light.

Sample Item:

“Sometimes the Moon looks like a full circle, sometimes it looks like a half circle, and sometimes it looks like a crescent (Set the Context). Explain why (Specify the Reasoning) the Moon appears to be different shapes at different times. You may use labeled drawings in your explanation (Point the Way).”

Released NAEP Item (http://nces.ed.gov)

Scoring Rubrics: Two Types

Generic Rubrics

“…define what content understanding looks like, in general, for any body of knowledge.”

“3” The response is clear, focused, and accurate.

“2” The response is clear and somewhat focused, but not compelling.

“1” The response misses the point, contains inaccurate information, or otherwise demonstrates lack of mastery of the material.

Classroom Assessment for Student Learning

Richard Stiggins, et. al. 2006

Generic Rubrics: Another Example:

Sophisticated: an unusually thorough, elegant, and inventive account; fully supported, verified, and justified; deep and broad: goes well beyond the information given.

In-Depth: an atypical and revealing account, going beyond what is obvious or what was explicitly taught; makes subtle connections; well supported by argument and evidence; novel thinking displayed.

Developed: reflects some in-depth and personalized ideas; going beyond the given – there is supported theory here, but insufficient or inadequate evidence and argument.

Intuitive: an incomplete account but with apt and insightful ideas; extends and deepens some of what was learned; some “reading between the lines”; account has limited support/argument/data or sweeping generalizations. There is a theory, but one with limited testing and evidence.

Naïve: a superficial account; more descriptive than analytical or creative; a fragmentary or sketchy account of facts/ideas or glib generalizations; a black-and-white account; less a theory than an unexamined hunch or borrowed idea.

Understanding by Design

Grant Wiggins & Jay McTight, 1998

Scoring Rubrics: Exercise-Specific

“Points are awarded when specific information appears in students’ responses.”



Examples: 5th Grade ScienceSometimes the Moon looks like a full circle,

sometimes it looks like a half circle, and sometimes it looks like a crescent. Explain why the Moon appears to be different shapes at different times. You may use labeled drawings in your explanation.

__________________________________________

__________________________________________

__________________________________________ALL examples taken from NAEP Released Items

http://nces.ed.gov/nationsreportcard

Exercise-specific Scoring RubricComplete (3) Student explanation includes all the points given

below. Student can provide a drawing correctly illustrating the phases of the moon. The Moon is visible because it reflects (or is illuminated

by) sunlight. The Moon revolves around the Earth. The portion of the illuminated half of the Moon that is

visible from Earth changes, thus making the Moon appear to change shape.

Partial (1-2) Student explains 1 or 2 aspects of the causes of the phases of the Moon without major misconceptions.

Unsatisfactory/Incorrect (0) Student does not correctly explain any aspect of the phases of the Moon, or explains aspects but includes major misconceptions.

Sample Student Response: Complete (3/3)

Sample Student Response: Partial (1/3)

Student Response: Unsatisfactory (0/3)

Examples: 7th Grade ScienceS.IP.07.15 Construct charts and graphs from data and

observations.

A student took a sample of water from a pond and examined it under a microscope. She identified several species of protozoans, including two species of Paramecium that are known to eat the same food. The student decided to examine the water sample every day for a week. She added food for the Paramecia each day and counted the number of each species. Her findings are summarized in the table below.

NUMBER OF PARAMECIA IN POND WATER SAMPLE

Day Species S Species T

1 50 50

2 60 80

3 100 90

4 150 60

5 160 50

6 160 30

7 160 20

Using the axes below, construct a graph showing the number of each species of Paramecium the student found each day. Be sure to label the axes.

Examples: 7th Grade Science

Exercise-specific Scoring Rubric

Complete (3): Student scales, plots, and labels the graph correctly.

Essential (2): Student scales and plots S and T correctly; one or both axis labels or plot labels are missing (or incorrect).

Partial (1): Student scales and plots either S or T correctly or scales and plots S and T combined (for example, adds data for each into one data point). No labels on graph.

Unsatisfactory/Incorrect (0): Student fails to plot data for either S or T correctly or produces an illogical graph.

Sample Student Response: Essential (2/3)

“Provides correct scales and plots for both sets of data, but lacks a label for the y-axis.”


“Provides correct scales and plots for both sets of data, but lacks labels for both axes.”


“Using different types of graphs, both responses scale and plot only the data for species S. They also both lack labels for the axes.”

Sample Student Response: Unsatisfactory

(0/3)

“Illogical graph of the data.”

Examples: High School Science

B1.2g Identify scientific tradeoffs in design decisions and choose among alternative solutions.

AND

B4.2h Recognize that genetic engineering techniques provide great potential and responsibilities.

“Some people believe that recombinant DNA technology has serious disadvantages. Describe one disadvantage that might result from the use of recombinant DNA technology. Then describe a plan or a policy for dealing with the disadvantage that could be followed by research scientists, doctors, public officials, or other people who are involved with recombinant DNA technology and its uses.”

Exercise-specific Scoring Rubric

Complete (3): Student response describes a reasonable disadvantage of recombinant DNA technology and provides a clear description of a plan for dealing with the disadvantage. Credited disadvantages (1 pt) include: Regulation of new strains Production of dangerous organisms Genetic Similarity - loss of diversity Regulation of applications/patents

Credited acceptable plans (2 pts) include: Informed consent Regulation Thorough testing Oversight committee

Essential (2): Student response describes a reasonable disadvantage and attempts a brief description of a plan for dealing with this disadvantage (e.g., test or observe, research further). OR Student response provides only a description of a plan.

Partial (1): Student response describes a reasonable disadvantage of genetic technology but does not develop a plan for dealing with the disadvantage.

Unsatisfactory/Incorrect (0): Student response states that there are no disadvantages, or states a disadvantage that is inaccurate or unreasonable.


“Student response states that making new kinds of viruses and mutations are a disadvantage, and outlines a plan that involves experimentation with human cells outside the body.”


“Student response discusses the production of dangerous viruses, and attempts a brief description of a plan.”

“Student response describes a disadvantage in general terms only, but does outline a plan that involves the setting of limitations.”



“Student response explains that the intermixing of genes could result in the production of nontreatable diseases. No plan is given.”


“Student response explains that the virus may have side effects, such as changing or altering a regular body function. No plan is given.”


(0/3)

“Student response does not state a clear disadvantage or outline a coherent plan.”


(0/3)

“Student response does not give a clear disadvantage, and states merely that these people should stop messing around with recombinant DNA technology.”

Activity #1: Write a Scoring Rubric

Question 1 – 4th Grade Earth Science

Question 2 – 5th Grade Life Science

Question 3 – 7th Grade Physical Science

Question 4 – 7th Grade Physical Science

Question 5 – High School Biology

Question 6 – High School Earth Science

E.ST.04.22 Explain that the spin of the Earth creates day and night.

“Everyone knows about day and night. Write what you think makes day and night. Draw a picture to show what you think.”

NAEP Released Items http://nces.ed.gov/nationsreportcard



“Everyone knows about day and night. Write what you think makes day and night. Draw a picture to show what you think.”

2 pts. The response indicates that the Earth turns so that the same face is not always facing the Sun. Example: “The Earth turns every 24 hours, and for 12 hours we are facing the Sun.”

1 pt. The response indicates that the Moon and Sun are on different sides of the Earth and the Earth rotates facing one and then the other. There is no implication that the Sun moves. Example: “In the day we face the Sun and in the night we turn to face the Moon.”

0 pt. The response indicates that the Sun moves (possibly across the sky) to cause night and day. Example: “The Sun moves and makes way for the Moon.” – PLUS –

1 pt. Diagram correctly illustrates Sun and Earth, with the side facing the Sun illuminated (day) and the side away from the Sun darkened (night).

Adapted from Classroom Assessment for Student Learning, Stiggins, et. al. 2006

L.OL.05.42 Explain how animal systems (digestive, circulatory, respiratory, skeletal, muscular, nervous, excretory, and reproductive) work together to perform selected activities.

Sample Item: “When you exercise strenuously, your body produces excess heat. Describe at least two things your body does to help prevent your temperature from rising excessively, and explain why the body's response is effective.”


Complete (3) Student indicates that the body keeps its temperature from rising through sweating or by blood vessels dilating AND states how these are effective – cooling by evaporation or transferring heat in blood to the air surrounding the skin.

Partial (1-2) Student indicates one or two methods the body keeps its temperature from rising but does not explain fully how the mechanisms work.

Unsatisfactory/Incorrect (0) Student provides little or no evidence of knowledge of any mechanism for losing heat during exercise.



S.IP.07.12 Design and conduct scientific investigations.

S.IP.07.13 Use tools and equipment (spring scales, stop watches, meter sticks and tapes, models, hand lens, thermometer, models, sieves, microscopes, hot plates, pH meters) appropriate to scientific investigations.

Sample Item: “Explain how (Specify the Reasoning) you can find out the volume of a solid object, such as a small rock (Set the Context), using only water and either a measuring cup or a graduated cylinder (Point the Way).”

Complete (3) Student correctly describes how to determine the volume of a solid object using water and a measuring cup or graduated cylinder. Comparison or change in volume of water should be explicit.

Method A: Pour some water into the graduated cylinder Record the water level (1 pt.)Then put the rock in the graduated cylinder Record the water level again (1 pt.)The difference between the first and second volume

measurements is the volume of the rock (1 pt.)

Method B:

Spillage Responses = Fill cup with water (to the top) (1 pt.), add rock, catch the water that overflows in separate container, and measure the overflow (1 pt.). Overflow = volume of the rock (1 pt.).

Partial (1-2) Student gives the initial steps involved in measuring the volume of a solid object, but does not compare or discuss the change in water level, volume of overflow, or volume of the rock.

Unsatisfactory/Incorrect (0) Student demonstrates no understanding of how to use water and a graduated cylinder to measure volume.


S.IA.07.11 Analyze information from data tables and graphs to answer scientific questions.

“One characteristic that can be used to identify pure metals is density. If you determine the density of a pure metal, you can determine what the metal is, as shown in the table below.

Suppose that you determine that a metal ring has a density of 15.3 grams/cm3. Assume that the ring is a mixture of some combination of the metals listed in the table. What can you determine about its composition from its calculated density? Explain your answer.”


Complete (3) Student response states that the ring is not pure gold but must contain some gold (1 pt.). Response may reason that the density of the ring (15.3 g/cm3) is less than the density of gold (19.3 g/cm3), but more than the density of any of the other metals (1 pt.). Response must also state that the identity of the other metals in the ring cannot be determined from the given information (1 pt.).

Partial (1-2) Student response identifies gold as being one of the metals in the mixture and may choose others based on such factors as averaging densities. Response does not state that the identity of the other metals in the ring cannot be determined from the given information.

Unsatisfactory/Incorrect (0) Student response does not identify gold as one of the metals in the mixture, or states that the ring is made of pure gold.


B4.3g Explain that cellular differentiation results from gene expression and/or environmental influence (e.g.,

metamorphosis, nutrition).

Sample Item: “Biologists know that nearly all cells in a person's body contain the same genes. For example, kidney cells contain the same genes as the cells that normally make hemoglobin. Given these facts, explain why kidney cells do not make hemoglobin even though they contain the hemoglobin gene.”


Complete (3) Student response states that even though all cells in a person's body contain the same genetic information (1 pt.), different cells "use" different parts of this information at different times (1 pt.), i.e., the gene for hemoglobin may be "turned on" in hemoglobin-manufacturing (e.g. bone marrow) cells, but "turned off" (1 pt.) in kidney cells.

Unsatisfactory/Incorrect Student response demonstrates little or no understanding of what causes different cells to perform different functions.



E1.2g Identify scientific tradeoffs in design decisions and choose among alternative solutions.

E2.4A Describe renewable and nonrenewable sources of energy for human consumption (electricity, fuels), compare their effects on the environment, and include overall costs and benefits.

“For each of the sources of electrical energy listed below, describe an advantage and a disadvantage of relying on that energy source for a large part of our country’s electrical energy.”

a. Solar b. Nuclear

c. Hydroelectric d. Fossil FuelsNAEP Released Items http://nces.ed.gov/nationsreportcard

Activity #1: Write a Scoring Rubric“For each of the sources of electrical energy listed below,

describe an advantage and a disadvantage of relying on that energy source for a large part of our country’s electrical energy.”

Source Advantage Disadvantage

Solar Less air pollution Expensive, Clouds interfere

Nuclear Less air pollution, large amount of energy per mass of fuel

Radioactive wastes, Possibility of accident or meltdown, Public acceptance

Hydroelectric Little pollution Limited access to rivers, Reservoirs flood land

Fossil fuels Abundant relatively cheap fuel Nonrenewable fuel source, Air pollution, Greenhouse gases

Complete (3) Includes 7-8 of the essential parts

Partial (2) Includes 5-6 of the essential parts

Unsatisfactory (1) Includes 3-4 of the essential parts

Incorrect (0) Includes less than 3 of the essential parts

Developing Constructed Response Assessments:

1. Plan the Assessment

2. Develop the Assessment

(including the Scoring Rubric!)

3. Critique the Assessment

4. Administer the Assessment

(Student Performance Data)

5. Revise the Assessment



Step 1: Plan the Assessment.

How Many Items? Aligned to which Content Expectations?

Grade Science Unit 1 Science Unit 2 Science Unit 3 Science Unit 4

5th Forces & Motion Animal Body Systems

Adaptations & Traits of

Organisms

Moon & Planets

6th Matter & Energy Ecosystems Soils, Rocks &

FossilsPlate Tectonics

& Earth’s Magnetic

Field

7th Energy & Waves Elements &

CompoundsCells, Plants &

Photosynthesis

Atmosphere, Weather & Climate,

Water Cycle

e.g. 2 CR Items per Unit = 8 Total

Step 1: Plan the Assessment.

How Many Items?

Aligned to which Content Expectations?

e.g. 2 CR Items per Unit = 24 Total Constructed Response Items per Physics Course

High School PHYSICS Units

1 Motion

2 Two-Dimensional Motion and Forces

3 Dynamics

4 Momentum

5 Periodic Motion

6 Mechanical Energy

7 Mechanical Waves

8 Electromagnetic Waves

9 Electric Forces

10 Electric Current

11 Energy Transformations

12 Energy and Society

Steps 2 & 3: Developing & Critiquing the

Assessment.

Eight Guidelines for Writing Constructed-Response Items:

1. Assess understanding beyond rote recall. There should be more than one way to answer a

question. There should be opportunities for students to

earn partial credit.

Harcourt Assessment, Inc. and

Michigan Department of Education, 2005

2. Keep the item within a reasonable scope. Avoid questions that are so broad that a

knowledgeable person could write multiple pages on the subject.

EXAMPLE:

Poor: Explain kinetic energy and gravitational potential energy.

Better: A pencil rolls across a tabletop and then falls to the floor. Describe the changes in the kinetic energy and gravitational potential energy of the pencil as it rolls, falls, and lands on the floor.


Assessment.

3. Define the task specifically. Don’t expect students to “read between the

lines.”

EXAMPLE:

Poor: Describe the differences between various types of rocks.

Better: Describe three differences between igneous and sedimentary rocks.


Assessment.

4. Break a complex task into parts. Makes an item more accessible to students. Put tasks in a logical sequence (first part is often at

lower cognitive level than later parts). Avoid redundancy.

EXAMPLE:

Poor: Juan and Valerie are designing an experiment to test whether a pesticide affects tomato plant growth. Identify four possible variables in this experiment. Choose one of these and explain how it can be controlled and how the results might change if it were not controlled.

Better: Juan and Valerie are designing an experiment to test whether a pesticide affects tomato plant growth. A. Identify four possible variables in this experiment. B. Choose one of these and explain how it can be

controlled.C. Explain how the results of their experiment might change

if this factor were not controlled.


Assessment.

5. Use verbs that discourage one-word responses. Avoid questions that can be answered simply

“yes” or “no.”

EXAMPLES:

“Explain” or “Illustrate” vs. “Name” or “List”


Assessment.

6. Use caution when asking subjective questions. Items should not intrude on student privacy. Do not ask students how they feel. Do not ask students to relate things to personal

experience. Any explanation or justification for a student’s

response should be based on the stimulus material.


Assessment.

7. Write the scoring rubric at the same time as the item. Include examples of “correct” or “partial”

responses.

8. Critique and confirm that the item elicits the intended response. Aligned to a specific Content Expectation. A common error is to ask one question, but base

the scoring rubric on an answer that really corresponds to another related question that goes into more depth than what is asked.

Harcourt Assessment, Inc. and Michigan Department of Education, 2005


Assessment.

Step 4: Administer the Assessment (Examine Student Data).

Sometimes the Moon looks like a full circle, sometimes it looks like a half circle, and sometimes it looks like a crescent. Explain why the Moon appears to be different shapes at different times. You may use labeled drawings in your explanation.

8th Grade NAEP Test, 2005 National Results

Complete 3%

Partial 18%

Unsatisfactory / Incorrect 76%

Omitted / Off Task 3%

NUMBER OF PARAMECIA IN POND WATER SAMPLE

Day Species S Species T

1 50 50

2 60 80

3 100 90

4 150 60

5 160 50

6 160 30

7 160 20

Using the axes below, construct a graph showing the number of each species of Paramecium the student found each day. Be sure to label the axes.

12th Grade NAEP Test, 2005 National Results

Complete 56%

Essential 23%

Partial 5%

Unsatisfactory / Incorrect 13%

Omitted / Off Task 4%


“When you exercise strenuously, your body produces excess heat. Describe what your body does to help prevent your temperature from rising excessively, and explain why the body's response is effective.”

2005 National Results

8th Grade 12th Grade

NAEP Test NAEP Test

Complete 2% 9%

Partial 66% 72%

Unsatisfactory /

Incorrect 23% 12%

Omitted / Off Task 9% 7%


Step 5: Revise the Assessment.

• Flaws in the assessment will become very clear.

• Can also see if and where instruction has “fallen short” and allow you to make improvements for your current and future sets of students.

Contact Information

Tom Wessels, DirectorGrand Traverse Regional Math and Science Center

Traverse Bay Area Intermediate School District

1101 Red Drive PO Box 6020

Traverse City, MI 49682

231.922.7875 [email protected]

writing constructed response assessment items:

Documents

constructed response

fixed response itemswhen

simple items

coordinated assessment

student learning

simple recall

simple hydrocarbons

simple checklist