using progress variables to map intellectual development cathleen a. kennedy and mark wilson

1

Using Progress Variables to Using Progress Variables to

Map Intellectual DevelopmentMap Intellectual Development

Cathleen A. Kennedy and Mark WilsonUniversity of California at Berkeley

[email protected]

2

OutlineOutline Purpose of the study Method

• BEAR Assessment System• Calibration with Common Item

Equating• Setting Standards

Findings • Progress Variables• Items• Progress Guides• Calibration• Performance Standards• Mapping Intellectual Development

Next Steps

Outline

3

Purpose of the StudyPurpose of the Study

Develop a framework for modeling intellectual development within a curricular unit:

1. Determine relevant constructs and instructionally useful performance levels

2. Construct formative assessments aligned with those levels

3. Establish performance expectations and associated cut-points

4. Demonstrate the use of graphical progress charts for formative feedback

Purpose

4

Context of the Study

Established FAST curriculum; 10-wk unit on Buoyancy Established m/c pre/post test Established instructional activities ≈ 300 students across 8 CA schools

Tasks: Gather more useful assessment data

1. Formalize data gathering in selected performance activities2. Extend some pre/post test items

Assess at least two learning dimensions: 1. A curriculum-specific content knowledge domain

2. A universal inquiry skill

Purpose

5

MethodMethod

Develop Assessments using BEAR* Assessment System• Determine Progress Variables• Design Items • Develop Progress Guides• Establish Measurement Model

Calibrate with Common Item Equating Set Performance Level Standards Data

Method

* Berkeley Evaluation & Assessment Research Center

6

BEAR Assessment System

Principle ToolA developmental perspective of learning Progress Variables

Alignment of assessment to Instruction Items Design

Useful to teachers Progress Guides

Evidence of quality and fairness Measurement Model

Method: BAS

Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.

7

Calibrate with Common Item Equating

Perf. Act. 1

Perf. Act. 2

Perf. Act. 3

Pretest Post Test

Method: Calibration

8

Standard Setting

Respondents with limited knowledge.

Define performance levels as logit

ranges.

Respondents with extensive knowledge.

Respondents with some knowledge.

Respondents with more knowledge.

Method: Standard Setting

9

?

?

?

Respondents with limited knowledge.

Respondents with extensive knowledge.

Respondents with some knowledge.

Respondents with more knowledge.

Define performance levels as logit

ranges.

Standard Setting

Method: Standard Setting

10

DataData

8 CA middle schools

1 teacher per school (most new to curriculum)

14 classrooms

220 students completed all assessments

• Calibration

• Student growth

75 additional students completed post tests

• Calibration

Method: Data

11

FindingsFindings1. Developed Two Progress Variables

• Buoyancy: WTSF (specific content)

• Reasoning (generic inquiry)

2. Modified existing assessment activities to elicit evidence of the two variables

3. Developed progress guides for the two variables

4. Calibrated Progress Variables

5. Set Performance Levels

6. Mapped Student Growth

Findings

12

BEAR Assessment System Principle 1BEAR Assessment System Principle 1 Classroom assessment system should be based

on a developmental perspective of student learning.

Building Block: Progress Variables*• Defines low, high and intermediate levels• Visual metaphor for

− how students develop and− how their responses change

Development of Progress Variables

Findings: Progress Variables

* Progress Variables are also referred to as Construct Maps in the BEAR Assessment System.

13

Development of Progress Variables

1. How many variables?

2. Which are most appropriate for the purpose?

3. How many levels are needed on each variable?


14

Candidate Progress Variables

Mass Volume Density

Graphing

Process InquiryBuoyancy

Why things sink & float

Building Explanations


15

Mass Volume Density

Graphing

Process InquiryBuoyancy:

WTSF

Why things sink & float

ReasoningBuilding

Explanations

Final Two Progress Variables


16

Useful Levels for Formative Feedback: WTSF

Instruct ional

Focus

Relative Density

Density: Objects

Density: Medium

Mass and Volume

Mass Volume

Introductionto the curriculum

1 2 3 4 5 6 7 8 9 10 11 12

Investigations


17

Levels of WTSF from the Curriculum

Understands mass or volume alone

Understands mass & volume

Understands density of matter

Understands relative density


18

More WTSF Levels after Reading Student Work

Misconceptions

Unresponsive

Understands mass or volume alone

Understands mass & volume

Understands density of matter

Understands relative density


19

BEAR Assessment System Principle 2BEAR Assessment System Principle 2

What is taught and what is assessed must be clearly aligned.

Building Block: Items Design • A framework for designing tasks to elicit specific

kinds of evidence about student knowledge, as described in one or more progress variables, seamlessly integrated into the instructional activities of a course.

Modification of Assessments

Findings: Items Design

20

Modification of Assessments

1. What is the purpose of assessing?

2. When should formal assessment take place?

3. What types of assessment tasks are appropriate?


21

Critical Junctures in Instruction: WTSF

Instruct ional

Focus

Relative Density

Density: Objects

Density: Medium

Mass and Volume

Mass Volume

Introductionto the curriculum

1 2 3 4 5 6 7 8 9 10 11 12

Investigations

Purpose is to determine readiness for next part of instruction.

RL4 RL7 RL10


22

Graphing Activity

Below is a data table and graph for the Sinking-Straws activity.

Depth of Sinking 4 cm 5 cm 6 cm 7 cm 8 cm

Mass of Straw and BBs

0.8 g 1.0g 1.2 g 1.4 g 1.6 g

Explain in detail what the data and graph tell you about mass and depth of sinking. Use information and evidence from the graph as well as information from what you know. You may write and draw a picture to show your explanation.


23

Predict-Observe-Explain Activity

We have three bottles. We know the mass of each bottle. We already measured their displaced volumes in water. Record the mass and displaced volume of each bottle in Table 1.

Table 1

Bottle # Mass (g) Displaced volume (ml)

Bottle 1

Bottle 2

Bottle 3

Based on what you know about mass and volume, in Table 2 PREDICT whether each bottle will float, sub-surface float, or sink when your teacher places the bottles into water. Explain why.

Table 2

Prediction Why do you think that the bottle will float, subsurface float, or sink?

Bottle 1

Float Sub-surface float Sink

Bottle 2


Bottle 3



24

Essay Activity

Explain below why things sink and float. Write as much information as you need to explain your answer. Use evidence and examples to support your explanation.


25

Multiple Choice with Justification Items

7. Amy put a wooden block into an overflow can filled with water. The block floats. She finds that the displaced water is 30 ml. What is the mass of the wooden block?

a. 10g

b. 30g

c. 60g

d. 90g

Please explain why you chose that answer:

___________________________________

___________________________________

___________________________________


26

Development of Progress Guides

Teachers are the primary managers and users of assessment data.

Building block: Progress Guides*• Categories of student responses must make sense to

teachers in the context of relevant progress variables.


Findings: Progress Guides

* Progress Guides are also referred to as the Outcome Space in the BEAR Assessment System.

27

Level “MV” on WTSF Progress Variable

Code Student knows Student needs Sample response

MV Mass and Volume

Student knows that floating depends on having a small mass and a large volume.

To progress to the next level, student needs to understand the concept of density as a way of combining mass and volume into a

single property.

“An object floats when its mass is small and

its volume is large.”

For each performance level, define:


28

Levels “RD” – “M,V” on WTSF

Level What the Student Already Knows What the Student Needs to Learn

RD

Relative DensityStudent knows that floating depends on having less density than the medium.“An object floats when its density is less than the density of the medium.”

DDensityStudent knows that floating depends on having a small density.“An object floats when its density is small.”

To progress to the next level, student needs to recognize that the medium plays an equally important role in determining if an object will sink or float.

MV

Mass and VolumeStudent knows that floating depends on having a small mass and a large volume.“An object floats when its mass is small and its volume is large.”

To progress to the next level, student needs to understand the concept of density as a way of combining mass and volume into a single property.

M V

MassStudent knows that floating depends on having a small mass.“An object floats when its mass is small.”

Volume Student knows that floating depends on having a large volume.“An object floats when its volume is large.”

To progress to the next level, student needs to recognize that changing EITHER mass OR volume will affect whether an object sinks or floats.


29

Level What the Student Already Knows What the Student Needs to Learn

PM

Productive MisconceptionStudent thinks that floating depends on having a small size, heft, or amount, or that it depends on being made out of a particular material.“An object floats when it is small.”

To progress to the next level, student needs to refine their ideas into equivalent statements about mass, volume, or density. For example, a small object has a small mass.

UF

Unconventional FeatureStudent thinks that floating depends on being flat, hollow, filled with air, or having holes.“An object floats when it has air inside it.”

To progress to the next level, student needs to refine their ideas into equivalent statements about size or heft. For example, a hollow object has a small heft.

OT

Off TargetStudent does not attend to any property or feature to explain floating.“I have no idea.”

To progress to the next level, student needs to focus on some property or feature of the object in order to explain why it sinks or floats.

NR No ResponseStudent left the response blank.

To progress to the next level, student needs to respond to the question.

Similar coding guide developed for Reasoning variable.


Levels “PM” – “NR” on WTSF

30

You have six blocks. Blocks 1, 2 and 3 are made of one material, while blocks 4, 5 and 6 are made of another material.

The density of blocks 1, 2 and 3 is 0.91 g/cm3, while the density of blocks 4, 5, 6 is 1.2 g/cm3.

For each block, predict whether it will sink, float or subsurface float in water. Give your reasons for your predictions.

3

Sink

Float

Subsurface Float

Block 3 will float because objects float when their density is less than 1.

WTSF – D

Reas. – P

Student Response

Reflective Lesson 10B


Two Codes

31

CalibrationCalibration

Classroom assessment requires reliability and validity evidence and evidence for fairness.

Building Block: Measurement model• Seeking interpretive quality• Multidimensional partial credit model in which

− Order of item difficulties is the same for all respondents− Order of respondents is the same for all item subsets− One parameter model for interpretive needs


Findings: Calibration

32

Respondent Location

Goal of Calibration: Interpretive Quality (1)

1. Person locations are interpreted in the context of item content (criterion referenced).

Buoyancy depends on the mass of the object.

Buoyancy depends on the object being flat, hollow,

filled with air, etc.

XRL4_A.Mass

RL4_A.ProductiveMisconceptions


33

Goal of Calibration: Interpretive Quality (2)

2. Student change derives meaning from items.

Buoyancy depends on the density of the object.

Buoyancy depends on the mass and volume of the

object.

Buoyancy depends on the mass of the object.

Buoyancy depends on the object being flat, hollow,

filled with air, etc.

RL4_A.Mass

RL4_A.ProductiveMisconceptions

XRL4_A.Mass&Volume

X


34

Calibrated All Items onto Same Scale

Five forms of post test contained all items from all instruments.

Anchored Pretest, RL4, RL7 and RL10 item difficulties from post test calibration.


35

Set Performance LevelsSet Performance Levels

Based on Thurstonian thresholds (50-50)

Find mean of thresholds for each step

Cut-point is midpoint between means

X

13.MV

6.MV

4.M

16.M

3.M

Findings: Set Perf. Levels

36

WTSF Performance Levels


37

WTSF Performance Levels


38

Buoyancy depends on the density of the object.

Buoyancy depends on the object being flat, hollow, filled with air, etc.

Buoyancy depends on the density of the object relative to the density of the medium.

Buoyancy depends on the mass and volume of the object.

Buoyancy depends on the mass or volume of the object.

1.4

0.4

-0. 6

-1.6

Relative Density

Density

Mass & Volume

Mass or Volume

Misconception

Set Criterion-Referenced Cut Points

Does not attend to any property or feature to explain floating

Unresponsive-2.3


39

Map Student PerformanceMap Student Performance

Class performance status

Individual student change

Comparisons to curricular expectations

Item diagnostics

Findings: Map Performance

40

Class Performance Status

Amy Tom Brian

After Lesson

7


41

Performance for Amy through Lesson 7

Pretest

Lesson 4

Lesson 7


42

Compared to Curricular Expectations


43

Diagnostic MapStudent - Brian

Variable - WTSFItem Set - RL7

Response Level Unachieved Levels

XXX

Item Diagnostics


44

Item DiagnosticsDiagnostic MapStudent - Brian



RL7b.MV RL7b.MorV XXX

RL7b.Mis

Performed as expected on this P-O-E item.


45

Item DiagnosticsDiagnostic MapStudent - Brian



RL7d.MV XXX RL7d.MorV RL7d.Mis

RL7d.UR

Performed more poorly than expected on this P-O challenge item.


46

Next StepsNext Steps1. Procedure tested in FOSS curriculum Fall 2006

2. Gathering teacher reactions to

a. Using standards-based graphs for formative feedback

b. Using software to record item codes

c. Consistency with intuition about students

Next Steps

47

Questions & AnswersQuestions & Answers

Thank You!

Cathleen Kennedy

[email protected]

using progress variables to map intellectual development cathleen a. kennedy and mark wilson

Documents