Using Classroom Vignettes to Measure Teachers’

Instructional Practices in Mathematics

Brian Stecher, Vi-Nhuan Le, Laura Hamilton, et al.

RAND

Mosaic II Project Overview

Research Question: What is the relationship between student achievement and exposure to reform-oriented curriculum and instruction?• In the context of NSF Local Systemic Change and

Systemic Reform Initiatives• Mathematics and science• 3-year longitudinal tracking of students• Grades 3-4-5 or 6-7-8• Multiple measures of teaching practices,

including vignette-based survey items• Multiple-choice and open-ended measures of

achievement

The Challenge of Measuring Instructional Practice

Common methods for measuring instructional practice have disadvantages that can limit their validity

METHOD DISADVANTAGES

Observations Complex, time consuming, expensive, subject to observer bias, difficult to summarize, etc.

Surveys Retrospective, memory and time distortions, subject to self-report bias, lack of shared understanding of reform terminology, etc.

Logs Limited scope, self-report bias, lack of shared understanding of reform terminology, etc.

Study a New Measure of Reform-Oriented Teaching

Develop vignette-based survey items

• Clear, situated descriptions of instructional events

• Core mathematics and science curriculum

• Admit reform-oriented teacher actions

Examine the validity of vignettes as a measure of instructional practice

• Consistency of teacher responses across “parallel” vignettes

• Relationship of vignettes with other measures of instruction, including surveys, logs, and observations

Focus on 4th grade mathematics

Rational for Vignette-Based Approach

Attempting to measure “intention” to engage in reform-oriented teaching based on responses to specific, hypothetical, familiar situations with clear behavioral alternatives.

ADVANTAGES DISADVANTAGES

• More realistic situations and choices

• Minimizes problems of reform terminology

• Represents a standardized situation

• Imposes standardized responses

• Greater reading burden

• Difficult to develop

• Verisimilitude?

• Unknown validity

Steps in Developing the Vignettes

Convened a panel of mathematics experts

Used NCTM standards and other documents to create a framework • Dimensions of reform-oriented mathematics

curriculum and instruction • Framed in behavioral terms

Selected two common 4th grade mathematics topics as setting

• Area/perimeter

• Two-digit multiplication

Format of the Vignettes

Instructions on how to respond

A context section specifying:• Length of the unit • Prior learning history of the students

Four separate but linked instructional problems: • Introducing the lesson• Responding to student error• Reconciling different approaches• Choosing learning objectives

Response options that represented a range of low- and high-reform actions• Likelihood of responding in this manner?

Designed to be “parallel” across the two topics

18. You are almost at the end of the unit on multiplying two-digit numbers. You ask students to work in pairs orgroups to solve the following problem.

Each school bus can hold 41 people. How many people can go on a field trip if the district has 23 busesavailable? Find the answer and demonstrate why it is correct.

After working on the problem for a while, you ask each group if they will share their work.

The first group says the answer is 943, and they use the standard algorithm to show how they obtained thisresult.

41

x 23__________ 123

82__________ 943

The second group says the answer is 943, and they explain that they broke the numbers into tens and onesand used the distributive property.

41 x 23 = (40 +1) x (20 + 3) = (40 x 20) + (40 x 3) + (1 x 20) + (1 x 3) = 800 + 120 + 20 + 3 = 943

After praising both groups for using eff ective strategies, how likely are you to do each of the following in responseto these two explanations?

(Circle One Response in Each Row)

Very unlikely Somewhatunlikely

Somewhatlikely

Verylikely

a. Ask the class if they can think of another way to solvethe problem

1 2 3 4

b. Suggest that the class check the results by using acalculator

1 2 3 4

c. Tell them the first group’s method is faster1 2 3 4

d. Tell them they are both right and move on to the nextproblem

1 2 3 4

e. Have a classroom discussion about the diff erencesbetween the two approaches

1 2 3 4

Setting Classroom Context

Imagine you are teaching mathematics to a fourth-grade class. The students are familiar with place value and with the multiplication facts up to 10 times 10. You are about to begin a week-long unit on multiplication involving larger numbers.

Introducing the Problem Scenario

You ready to start the unit by introducing multiplication of two-digit numbers. How likely are you to do each of the following as an initial activity for the unit?

• Ask students what they know about multiplying two-digit numbers

• Have students work in groups to solve problems such as finding the number of inches in 20 feet

• Demonstrate the standard algorithm for multiplying 2 two-digit numbers

Note: Options Continue

Responding to Student Error Scenario

Part way through unit… Students know standard algorithm…

You ask your students to figure out how many miles Jerry can drive on 17 gallons of gas if Jerry’s car goes 23 miles per gallon of gas.

You ask Leah if she will share her solution. She says “I multiplied 20 times 20 and got 400. This works because 23 is three more than 20 and 17 is three less than 20.” How likely are you to do the following?

• Ask Leah if subtracting 3 miles per gallon is the same as adding 3 gallons of gas

• Explain that 23 x 17 is actually 391. Then pose another similar problem

Note: Options Continue

Reconciling Different Approaches Scenario

Almost at end of unit…

You ask students to work in groups to find how many people can go on a field trip if the district has 23 buses available, and each school bus can hold 41 people.

The first group says the answer is 943, and uses the standard algorithm [figure]. The second group gives the same answer, but explain that they broke the numbers into tens and ones and used the distributive property [figure]. How likely are you to do the following?

• Have a classroom discussion about the differences between the two approaches

• Tell them that the first group’s method is faster Note: Options Continue

Choosing Learning Objectives Scenario

If you were to design a unit on two-digit multiplication, how much emphasis would you place on each of the following objectives?

• Students will understand how place value relates to two-digit multiplication

• Students will be able to use place-value blocks or graph paper to represent two-digit multiplication

• Students will be able to define the terms ‘factor’ and ‘product’

Note: Options Continue

Study Methods

Sample

• 80 fourth-grade teachers in 20 schools from a district participating in the Local Systemic Change program

• Observed 39 teachers from 13 schools

Measures of instruction

• Surveys

• Logs

• Vignettes

• Classroom observations

Creating Vignette-Based Scores of Reform-Oriented Teaching

Assigned a value to each response option (low- to high-reform)

Create two scales

• Reform Inclinations

• Standardized sum of teachers’ answers to the high-reform response options (N = 27 items)

• Euclid

• Euclidean distance of each teacher from an “ideal” high reform teacher (N = 51 items)

.80)(á =

.86)(á =

Distribution of Vignette-Based Score of Reform Teaching (Euclid)

Other Measures from Survey, Logs and Observation

Processes: Extent to which teachers emphasized NCTM-endorsed cognitive processes (N = 5 items)

Reform: Frequency with which the teachers engage in reform-oriented practices (N = 9 items)

Discussion: Amount of time the class engaged in dialogue about mathematical thinking and understanding (N = 4 items)

Seatwork: Extent to which students engaged in seatwork and other low-reform activities (N = 2 items)

Understanding: Extent to which the teacher facilitated deeper mathematical understanding (N = 5 items)

Findings Regarding Vignette-Based Measures

Responses are stable across “parallel” mathematical contexts

“Reform Inclination” scale correlates with several survey and log measures (but not observational measures)

“Euclidean” scale correlates with observational measures (but not survey and log measures)

Mixed evidence regarding validity of vignette-based scales

Next Steps

Examine relationships of vignette-based measures with student outcomes, particularly achievement

Improve the quality of vignettes through “think aloud” interviews with teachers

Investigate how length and level of detail in vignettes affect how teachers respond

Explore the applicability of vignettes in other contexts