1 prepared for irving m.s. on may 10, 2006 using data project overview collaboration between terc...
TRANSCRIPT
1
Prepared for Irving M.S. on May 10, 2006
USING DATA PROJECT OVERVIEW• Collaboration between TERC and WestEd
• Funded by the National Science Foundation
• Based on Using Data/Getting Results
• Working with mathematics and science projects nationally to improve teaching and learning
• Provides professional development and materials
2
TODAY’S PLAN
• 100% of participants will understand the Data Dialogue Process as measured by growth on a
consensogram
3
TO DO LIST
Participants will:• Understand the data dialogue process
• Identify the benefits and limitations of aggregated, disaggregated, and strand data
• Apply the data dialogue process• Discuss ideas for implementation
4
THE USING DATA PROCESS/PDSA
5
Achieve
0
5
10
15
20
25
30
35
40
45
50
Crenshaw Hartford Lehman Souers
2002-20032003-2004
StudentLearning
Goal
Generate Solutions
Strategies Outcomes
Strategies Outcomes
Causes
Verify
StudentLearningProblem
Identify
Foundation
Build
DATA-DRIVEN DIALOGUE
6
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
THE DATA DIVIDE
7
8
What do you think bridges the gap between data and results?
What are the features of a culture that supports data-driven
dialogue?
Partner Talk
COLLABORATIVE CULTURE IS FIRST ORDER OF BUSINESS….
• Creating a collaborative culture is the single most important factor for successful school improvement initiatives and the first order of business for those seeking to enhance the effectiveness of their schools.
Eastwood and Louis (1992). Restructuring That Lasts: Managing the Performance Dip. Journal for School Leadership 2 (2), 213-224.
9
NEED FOR A COLLABORATIVE CULTURE
• Throughout our ten-year study, whenever we found an effective school or an effective department within a school, without exception that school or department has been a part of a collaborative professional learning community. Milbury McLaughlin, 1995. Creating Professional Learning Communities. Kenote address presented at NSDC Conference, Chicago, IL
10
USING DATA PROJECT: BUILDING THE BRIDGE BETWEEN DATA AND RESULTS
11
Using DataProfessional
Development
Instructional ImprovementData UseCollaborationLeadership &
Capacity
Culture/Equity
USING DATA PROJECT ASSUMPTION
12
Collaborative inquiry – school teams constructing meaning of student learning
problems and testing out solutions together through rigorous use of data and reflective dialogue – unleashes the resourcefulness of educators to solve the biggest problems
schools’ face.
USING DATA PROJECT ASSUMPTION
13
Significant improvement in mathematics learning and closing achievement gaps is a moral responsibility and a real possibility in
a relatively short amount of time - one to three years.
POSSIBILITY
14
Poverty, Ethnicity, and Achievement
Poverty and Ethnic Minority Enrollment
% o
f S
tud
en
ts P
rofi
cie
nt
or
Ab
ov
e
Source: Accountability in Action by Douglas Reeves, Center for Performance Assessment, Denver, Colorado www.makingstandardswork.com/ResourceCtr/books
XSome high-poverty,
high-minority schools are achieving at this level
UDP SCHOOLS OUTPERFORM CONTROLS
15
% Proficient 6th Grade Science Students in Tennessee County, State 2004-2005 TCAP
50
55
60
65
70
75
80
85
90
95
Interactions Living /Environment
Food Production /Energy for Life
Diversity / LivingThings
Biological Change Earth in theUniverse
Energy
%
Treatment School 1Treatment School 2Control School 1Control School 2
Source: Personal Communication, 2005
MATHEMATICS GAINS IN CANTON CITY:OHIO PROFICIENCY TEST
16
0
5
10
15
20
25
30
35
40
45
50
Crenshaw Hartford Lehman Souers
2002-20032003-2004
Per
cent
age
Pas
sing
33%
42%
20%
35%
24%
47%
20%
36%
Middle School
Sixth Grade Proficiency Results
Source: Ohio Department of Education
MATHEMATICS GAINS IN CANTON CITY: SIXTH GRADE QUARTERLIES
17
50.2
57.3
43.0
49.7
40.7
52.2
0
10
20
30
40
50
60
Pe
rce
nt
Co
rre
ct
Decimals Fractions Total
2003-20042004-2005
District Sixth Grade
ChunkSource: Canton City School District
MATHEMATICS GAINS IN CANTON CITY: SEVENTH GRADE QUARTERLIES
18
35.0
43.8
31.7
40.1
35.0
40.2
0
10
20
30
40
50
Pe
rce
nt
Co
rre
ct
Computation Algebra Total
2003-20042004-2005
District Seventh Grade
ChunkSource: Canton City School District
MATHEMATICS GAINS IN CANTON CITY: EIGHTH GRADE QUARTERLIES
19
43.146.8
44.3 45.3
29.131.7
35.238.9
0
10
20
30
40
50
Per
cent
Cor
rect
Computation ScientificNotation
Proportions Total
2003-20042004-2005
District Eighth Grade
ChunkSource: Canton City School District
JOHNSON COUNTY IMPROVES MATHEMATICS - GRADES 3, 5, 8
20
7788
72
86
36
73
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
Perc
en
t P
rofi
cie
nt
All Low SES SWD
CRT Proficiency: Mathematics Grades 3, 5, 8
20042005
390
420
238
297
28
56
Source: TN DOE
Number on column = number of students
JOHNSON COUNTY IMPROVES MATHEMATICS - GRADES 9 -12
21
84 8680 83
30
58
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
Perc
en
t P
rofi
cie
nt
All Low SES SWD
CRT Proficiency: Mathematics 9 - 12
200420051
3156
63
83 1
112
Source: TN DOE
Number on column = number of students
22
What do you think are the changes in school culture that are taking
place in the schools that are improving results?
Table Talk
INSTRUCTIONAL IMPROVEMENT SHIFT
23
Data to sort, opportunities for
some
Less Emphasis More Emphasis
Data to serve, opportunities for all
SORTING STUDENTS
24
Pe
rce
nt o
f Stu
den
ts
Percentile
VERY DUMB
SORTA SMART
VERY SMART
SORTA DUMB
DOING THINGS DIFFERENTLY
26
”“The data are causing us to do things differently.
We set a goal for improvement. Now we teach to achieve that goal.
— Mia Merrick, Teacher
Desert Eagle Secondary School
DATA USE SHIFT
27
Carrot and stick, avoidance
Feedback for continuous improvement, frequent and in depth use
Less Emphasis More Emphasis
SHIFTS THAT ARE MOVING SCHOOLS FROM RESIGNATION TO
EMPOWERMENT
28
Culture
Instructional Improvement
Collaboration
Data Use
Leadership & Capacity
Less Emphasis More Emphasis
External accountability Internal and collective responsibility, equity
Data to sort, opportunities for some
Data to serve, opportunities for all
Top-down, premature data-driven decision
making
Ongoing data-driven dialogue and collaborative inquiry
Carrot and stick, avoidance
Feedback for continuous improvement, frequent and in depth use
Individual charismatic leaders as change agents
Learning communities with many change agents
AFFINITY CHART ACTIVITY
• Reflect on the culture of your school• What shifts need to occur to move towards a
more collaborative, data-driven culture?• Use post-its to record your ideas• One idea per post-it• Place them all on the flip chart when your done
29
LESSONS: COLLABORATIVE INQUIRY WORKS IF…
• High-quality professional development• A clear structure and process is provided for data-
driven dialogue and sense-making• Skilled data teams - data literacy, content and
pedagogy, equity - and data facilitators• Collective response-ability for student learning• Time for data teams to work - weekly• Timely access to robust data - benchmark
assessments, item level data, student work• District and school administrative support
30
TYPES OF STANDARDIZED ASSESSMENTS: NORM-REFERENCED TESTS (NRT)
• Compare performance of individuals and groups• Rank and sort students by comparing them to a
national “norm” group• Examples: Stanford 9, Iowa Test of Basic Skills,
Terra Nova
31
SORTING STUDENTS
32
Pe
rce
nt o
f Stu
den
ts
Percentile
VERY DUMB
SORTA SMART
VERY SMART
SORTA DUMB
TYPES OF STANDARDIZED ASSESSMENTS: CRITERION-REFERENCED TESTS (CRT)
• Interpret performance of individuals and groups in relation to a set of criteria, standards, or benchmarks
• Determine student mastery of the criteria• Used to make instructional and programmatic
decisions (e.g., curriculum)• Examples: most state assessments,
New Standards Reference Exam
33
FUNDAMENTAL BELIEF
“All kids can learn so we establish high standards that we expect all students to achieve.”
34
SERVING STUDENTS
35
A - Advanced
P - Proficient
NI - Needs Improvement
W - Warning
36
“Stop asking me if we’re almost there!We’re nomads, for crying out loud!”
DATA-DRIVEN DIALOGUE
37
PHASE 1Predict
PHASE 3Infer/Question
PHASE 2Observe
GoVisual
• With what assumptions are we entering?
• What are some predictions we are making?
• What are some questions we are asking?
• What are some possibilities for learning that this experience presents us with?
Surfacing experiences, possibilities, expectations
• What important points seem to “pop out”?
• What are some patterns or trends that are emerging?
• What seems to be surprising or unexpected?
• What are some things we have not explored?
Analyzing the data
• What inferences and explanations can we draw?
• What questions are we asking?
• What additional data might we explore to verify our explanations?
• What tentative conclusions might we draw?
Generating possible explanations
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
38
NEVAZOH
• Take a minute and read H4.3• Highlight key points about Nevazoh
39
IDENTIFY STUDENT LEARNING PROBLEM: DRILL DOWN DEEP
40
StudentLearningProblem
Aggregate / Summary Reports
Disaggregated Results
Cluster / Strand / Subscale
Item Analysis
Student Work
Triangulate Student Learning Data
1 2 3
BACKGROUND ON NEVAZOH
• Grade 6-8 urban school in Midwestern state• 724 students
• 62% economically disadvantaged• 35% African American, 57% white, 6% multi-racial
• At Risk School – 42% of 6th graders proficient in science in 2004-2005, 33% in mathematics, 51% in reading
• The school did not meet adequately yearly progress and has been placed on “Academic Watch.”
41
THREE-YEAR AGGREGATE DATA TRENDS:SIXTH-GRADE MATHEMATICS FROM H4.5
42
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
61 61
53
44 47
3336
3125
0
10
20
30
40
50
60
70
80
90
100
2001 - 2002 2002 - 2003 2003 - 2004
State
School
District
Percentage of students at and above proficiency
AGGREGATE DATA
• What are the benefits of aggregate data?• What limitations do these data have?
43
IDENTIFY STUDENT LEARNING PROBLEM: DRILL DOWN DEEP
44
StudentLearningProblem
Aggregate / Summary Reports
Disaggregated Results
Cluster / Strand / Subscale
Item Analysis
Student Work
Triangulate Student Learning Data
1 2 3
DATA-DRIVEN DIALOGUE
45
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
PHASE 1: PREDICT STARTERS
46
I predict…
I assume…
I wonder…
I’m expecting to see…
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
THREE-YEAR DISAGGREGATED DATA: PHASE 1, PREDICT
• What do you think the disaggregated data will represent?
• What, if any, achievement gaps do you expect to see reflected in the data?
• Document your predictions on chart paper.
47
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
GO VISUAL
• Create a graph or graphs to display the disaggregated data from H4.6
48
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
49
6169
74 77
22 2128
3232 3439 41
0
10
20
30
40
50
60
70
80
90
'90 '92 '96 '00
%White
Black
Hispanic
ExampleExamplePercentage At or Above Basic Proficiency in Mathematics by Percentage At or Above Basic Proficiency in Mathematics by
Race/Ethnicity- NAEP 1990-2000Race/Ethnicity- NAEP 1990-2000
DATA-DRIVEN DIALOGUE
50
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
DATA-DRIVEN DIALOGUEPHASE 2: OBSERVE
• What important points seem to pop out?• What is surprising, unexpected? • What gaps do you observe? • Always document your observations on chart
paper.
51
Adapted from Lipton and Wellman, 1999
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
OBSERVATION REMINDERS
• Made by the five senses• Quantitative and qualitative• Contain no explanations
52
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
53
BECAUSE
CONCEPT ATTAINMENT TESTERS
• Our teachers aren’t emphasizing basic skills enough
• 45% of our eighth graders are not meeting the standard in computation
• Teachers aren’t teaching inquiry-based science because they feel too much pressure to cover the curriculum
• On a recent survey, a majority of elementary teachers reported that they needed more professional development in science content
54
NOW TAKE 10 MINUTES TO DOCUMENT YOUR OBSERVATIONS
55
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
REFINING OBSERVATIONS• Does each statement communicate a
single idea about student performance?• Are the statements short and clear?• Do the statements use everyday language that
is easy to understand?• Do the statements incorporate numbers or
phrases that quantify data? • Are the statements consistent with the way in
which the data are reported?• Now, go back and put a star next to the
predictions that were confirmed
56
Source: The ToolBelt: A Collection of Data-Driven Decision-making Tools for Educators. Copyright© 2004 Learning Point Associates. All rights reserved. For a more complete account of this process, see http://www.ncrel.org/toolbelt/tools.htm
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
57
What’s the final step in the data dialogue process?
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
DATA-DRIVEN DIALOGUE
58
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
DISAGGREGATED DATA:PHASE 3, INFER
• What questions do you have about the data?• What school factors could be contributing to
lowered student learning?• Why do the gaps exist?• Document your inferences on chart paper.
59
DISAGGREGATED DATA: CAUTIONS
• Look at trends over time• Consider sample size. Small numbers within
subgroups can lead to wide variation in results• Consider more than one statistic – not just
percentage proficient and above• Improve program for all students
60
STUDENT LEARNING: UDP CORE VALUE
61
We are committed to acting consistently with our belief that all students can learn and to
being active anti-racists in our schools and our own work. We are committed to working with schools to close achievement gaps within one
to five years.
STUDENT LEARNING: UDP CORE VALUE (CONTINUED)
62
We believe that closing achievement gaps is not only a moral responsibility, it is a
very real possibility — within one to five years of implementing a powerful professional
development program based on collaborative inquiry.
THE DRIVING QUESTION:
•What does race, class and gender have to do with it?
63
64
“. . .very little will change for these disenfranchised students unless we directly confront racial, class, cultural and gender biases and the inequitable practices they spawn. Reform that does not put equity center stage has not and will not bring about high levels of mathematics and science achievement all.”
Nancy Love
Powerful Words
IDENTIFY STUDENT LEARNING PROBLEM: DRILL DOWN DEEP
65
StudentLearningProblem
Aggregate / Summary Reports
Disaggregated Results
Cluster / Strand / Subscale
Item Analysis
Student Work
Triangulate Student Learning Data
1 2 3
STRAND DATA ANALYSIS:QUESTIONS TO CONSIDER
• What content strands are the test items actually measuring? What standards or learning outcomes are being tested?
• What are areas of relative strength and weakness in our students’ performance on content strands?
• Ideally, look at subgroups of student performance in strand areas: How do subgroups of students perform in the strands? Are there achievement gaps among student subgroups?
66
67
What’s the first step in the data dialogue process?
DATA-DRIVEN DIALOGUE
68
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
STRAND LEVEL DATA: PHASE 1, PREDICT
• Use Handout H5.3 to help you predict for each strand.
• In which strand areas do you expect to have the lowest percentage of proficient students? The highest?
• In which levels of understanding do you expect to have the lowest percentage of proficient students? The highest?
• Document your predictions on chart paper.
69
70
What’s the next step in the data dialogue process?
To go visual, use handout H5.7 for Nevazoh’s strand data and create and vertical plot.
STRAND DATA GO VISUAL EXAMPLE:VERTICAL PLOT
71
CRT Content (insert your terminology, e.g., strand, cluster, subscale) ______________ AnalysisSchool____________Assessment________Year____Content Area______________
Patterns, relations, functions63%
60% Problem-solving strategies
38% Number and number relations
100
90
80
70
60
50
40
30
20
10
0
Percentage of students at and above proficiency
STOPLIGHT HIGHLIGHT: MULTIPLE CHOICE ITEM ANALYSIS DATA
72
Adapted from NCREL, Data Retreat Facilitator Guide, 2001
Highlight Color
Green
Yellow
Pink
Meaning
Go! Meets expectations
Caution! Below expectations
Urgent! In immediate need of improvement
% Correct (our cutoffs)
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
70-100
50-69
Below50
73
After we go visual what do we do?
DATA-DRIVEN DIALOGUE
74
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
STRAND LEVEL DATA:PHASE 2, OBSERVE
• Which strand areas have more students at proficiency than above proficiency?
• Which strand areas are in the “red zone”?• Which strand areas are in the “green zone”? • Document your observations on chart paper.
75
76
What’s the final step in the data dialogue process?
DATA-DRIVEN DIALOGUE
77
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
STRAND LEVEL DATA:PHASE 3, INFER
• Looking at all of the data (proficiency percentages, learning objectives, and number of items on the test), what do you think students do and do not understand and know about mathematics?
• What school factors could be contributing to lowered student learning?
• Document your inferences on chart paper.
78
WHAT CAN WE LEARN FROM STRAND DATA?
• Specific information about learning objectives or content areas that students do and do not know or understand
• Provide information for further investigation — if only three or four test items within a strand, we need to look at more information
• Disaggregate! If you have access to disaggregated strand data, you have access to a wealth of information!
79
STRAND DATA: CAUTIONS
• Different tests define strands differently. Don’t assume that strands that are named similarly measure the same skills or understanding
• A small number of items within any given strand can skew results
• Strand analysis is most useful when combined with item and student work analysis
80
FOCUS ON: NEVAZOH STRAND DATA
• Review observations and inferences on the strand (cluster) data
• Looking at all of the data (proficiency percentages, learning objectives, and number of items on the test), what do you think students do and do not understand and know about mathematics?
81
STAND AND DELIVER REFLECTION
82
I understand…
I wonder…
I am struck by…
IDENTIFY STUDENT LEARNING PROBLEM: DRILL DOWN DEEP
83
StudentLearningProblem
Aggregate / Summary Reports
Disaggregated Results
Cluster / Strand / Subscale
Item Analysis
Student Work
Triangulate Student Learning Data
1 2 3
ITEM ANALYSIS FOR NU - PART 1
84
TestPart
Strand Outcome#
Item#
CorrectAnswer
A--0
B--1
C--2
D--3 4
Blank Bldg%
Dist%
State%
N =Bldg
M NU 10 1 C 15 23 49 12 0 0 49 48 62 282M NU 10 7 S 47 32 12 0 0 10 12 9 20 282M NU 06 16 D 4 7 4 85 0 0 85 81 87 282M NU 07 19 S 47 15 28 0 0 9 28 23 45 282M NU 08 20 D 6 27 11 56 0 0 56 59 73 282M NU 06 22 B 16 38 26 20 0 1 38 37 53 282
M NU 09 35 D 50 0 33 17 0 0 17 25 40 282M NU 08 41 D 9 12 39 37 0 3 37 27 44 282M NU 06 45 B 33 50 8 4 0 6 50 48 58 282
Nevazoh 6th-Grade Mathematics 2004-2005 CRT AssessmentItem Data - Percent Correct
Answers reported in percent
Correct Answer Column: S = short answer E = extended response
Data is for illustration only. Source: Ohio Department of Education, www.ode.state.oh.us (permission pending)
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
STOPLIGHT HIGHLIGHT: MULTIPLE CHOICE ITEM
ANALYSIS DATA
85
Adapted from NCREL, Data Retreat Facilitator Guide, 2001
Highlight Color
Green
Yellow
Pink
Meaning
Go! Meets expectations
Caution! Below expectations
Urgent! In immediate need of improvement
% Correct (our cutoffs)
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
70-100
50-69
Below50
ITEM ANALYSIS FOR NU - PART 1
86
Test Part
Strand Outcome #
Item #
Correct Answer
A -- 0
B -- 1
C -- 2
D -- 3
4
Blank Bldg %
Dist %
State %
N = Bldg
M NU 10 1 C 15 23 49 12 0 0 49 48 62 282 M NU 10 7 S 47 32 12 0 0 10 12 9 20 282 M NU 06 16 D 4 7 4 85 0 0 85 81 87 282 M NU 07 19 S 47 15 28 0 0 9 28 23 45 282 M NU 08 20 D 6 27 11 56 0 0 56 59 73 282 M NU 06 22 B 16 38 26 20 0 1 38 37 53 282
M NU 09 35 D 50 0 33 17 0 0 17 25 40 282 M NU 08 41 D 9 12 39 37 0 3 37 27 44 282 M NU 06 45 B 33 50 8 4 0 6 50 48 58 282
Nevazoh 6th-Grade Mathematics 2004-2005 CRT AssessmentItem Data - Percent Correct
Answers reported in percent
Correct Answer Column: S = short answer E = extended response
Data is for illustration only. Source: Ohio Department of Education, www.ode.state.oh.us (permission pending)
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
ITEM ANALYSIS - PART 2
87
Test Part
Strand Outcome #
Item #
Correct Answer
A -- 0
B -- 1
C -- 2
D -- 3
4
Blank Bldg %
Dist %
State %
N = Bldg
M NU 10 1 C 15 23 49 12 0 0 49 48 62 282 M NU 10 7 S 47 32 12 0 0 10 12 9 20 282 M NU 06 16 D 4 7 4 85 0 0 85 81 87 282 M NU 07 19 S 47 15 28 0 0 9 28 23 45 282 M NU 08 20 D 6 27 11 56 0 0 56 59 73 282 M NU 06 22 B 16 38 26 20 0 1 38 37 53 282
M NU 09 35 D 50 0 33 17 0 0 17 25 40 282 M NU 08 41 D 9 12 39 37 0 3 37 27 44 282 M NU 06 45 B 33 50 8 4 0 6 50 48 58 282
Nevazoh 6th-Grade Mathematics 2004-2005 CRT AssessmentItem Data
Answers reported in percent
Correct Answer Column: S = short answer E = extended response
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
Data is for illustration only. Source: Ohio Department of Education. www.ode.state.oh.us (permission pending)
STOPLIGHT HIGHLIGHT: MULTIPLE CHOICE ITEM DATA - DISTRACTOR
PATTERNS
88
Highlight Color Meaning % Correct
(our cutoffs)
Pink Urgent! In immediate need of improvement
Adapted from NCREL, Data Retreat Facilitator Guide, 2001
Highlight high-frequency INCORRECT selections.
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
ITEM ANALYSIS - PART 2
89
Test Part
Strand Outcome #
Item #
Correct Answer
A -- 0
B -- 1
C -- 2
D -- 3
4
Blank Bldg %
Dist %
State %
N = Bldg
M NU 10 1 C 15 23 49 12 0 0 49 48 62 282 M NU 10 7 S 47 32 12 0 0 10 12 9 20 282 M NU 06 16 D 4 7 4 85 0 0 85 81 87 282 M NU 07 19 S 47 15 28 0 0 9 28 23 45 282 M NU 08 20 D 6 27 11 56 0 0 56 59 73 282 M NU 06 22 B 16 38 26 20 0 1 38 37 53 282
M NU 09 35 D 50 0 33 17 0 0 17 25 40 282 M NU 08 41 D 9 12 39 37 0 3 37 27 44 282 M NU 06 45 B 33 50 8 4 0 6 50 48 58 282
Nevazoh 6th-Grade Mathematics 2004-2005 CRT AssessmentItem Data - Distractor Patterns
Answers reported in percent
Correct Answer Column: S = short answer E = extended response
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
Data is for illustration only. Source: Ohio Department of Education. www.ode.state.oh.us (permission pending)
17
DATA-DRIVEN DIALOGUE
90
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
91
BECAUSE
DATA-DRIVEN DIALOGUEPHASE 2: OBSERVE
• What important points seem to pop out?• What patterns or trends are emerging?• What is surprising, unexpected?• What questions do we have now?• How can we find out?• Document your observations
92
Adapted from Lipton and Wellman, 1999
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
DATA-DRIVEN DIALOGUE
93
PHASE 1Predict
PHASE 3Infer/Question
Adapted from Organizing Data-Driven Dialogue by Laura Lipton Bruce Wellman, MiraVia LLC, 2001
PHASE 2Observe
GoVisual
ITEM-LEVEL DATA: PHASE 3, INFER
• Examine frequently missed (pink) items and those with high-frequency incorrect (pink) responses
• What inferences and explanations might we draw as to why students are missing these items? Choosing the distractors?
• Record inferences on data wall• What additional data do we want to collect?• What questions do we have now?
94
PHASE 1 Predict
PHASE 3 Infer/Question
PHASE 2 Observe
GoVisual
IDENTIFY STUDENT LEARNING PROBLEM: DRILL DOWN DEEP
95
StudentLearningProblem
Aggregate / Summary Reports
Disaggregated Results
Cluster / Strand / Subscale
Item Analysis
Student Work
Triangulate Student Learning Data
1 2 3
USES AND PURPOSES OF STUDENT WORK
• Discuss with a partner:• For what purposes have you looked at
student work?• What protocols have you used?
96
STUDENT WORK IN THE UDP PROCESS
• To triangulate and verify the student learning problem identified through CRT drill-down
• To better understand student thinking in relation to the identified problem
97
PUTTING IT ALL TOGETHER
99
Aggregate Data All content areas, over time
Disaggregated DataAll content areas, all subgroups of students, over time
Strand DataStrands for all content areas, all subgroups of students, over time
ItemAnalysis of numerous items, disaggregated, all content areas, over time
Student WorkNumerous samples, disaggregated, over time
HOW WILL THE UDP WORK AT MY SCHOOL?
• What kinds of things would you have to have in place to do this type of work effectively?• Put 5 things or ideas on post-its and post
• How can Irving implement this next year?• How can you help lead this type of dialogue at
your school?• How will you begin to use the data dialogue
process?• What are the ramifications of waiting another
year?
100
EVALUATIONS
101
Good luck!
Proceed with passion
and persistence!