venessa a. keesler, ph.d. bureau of assessment and accountability michigan department of education...
TRANSCRIPT
Venessa A. Keesler, Ph.D.
Bureau of Assessment and Accountability
Michigan Department of Education
Presentation to MASFPS Fall Directors’ Institute
October 4, 2012
#1: NO ONE (and I do mean no one) likes accountability.
#2: If you are the one who is holding people accountable, you are not going to be the most popular person in the room
#3: See point number 1
Things I’ve learned about accountability since taking this job
Three myths; one reality•Myth #1: To drive reform•Myth #2: To create education policy•Myth #3: Because we are gluttons for punishment
Reality: • Accountability metrics/systems are quantitative articulations of the core policy beliefs of the education system• They help us measure our progress in meeting those core policy goals• They are the measure, not the purpose or the goal
So why do we do accountability?
A new era of accountability •Switching from a purely criterion-based system to a normative system•Criterion-based systems: Set average proficiency targets for schools.•Normative system: identifies the “worst” or “best” or “lowest” or “highest”
Accountability Landscape: 2012
Policy imperative for NCLB: all students CAN and SHOULD demonstrate proficiency criterion-system with proficiency targets for all schools and subgroups
10 years later: our average achievement is increasing, but we still have students and schools lagging behind
New policy imperative (ESEA Flex): we must target our lowest performing schools AND our lowest performing students more specifically and strategically
Why the change?
“Never trust an average”
Averages mask low performance
Example: • Proficiency target is 50%• Sunshine School has 20 students, 10 of whom are proficient and 10 of whom are not.• Sunshine School meets it’s target; hooray!• BUT those 10 not proficient students are possibly left behind
The problem with average proficiency rates at a school or district level
Top to Bottom Ranking• Priority Schools (bottom 5%)• Focus Schools (largest achievement gaps)• Reward Schools (high performing, high progress, beating the odds)
Accountability Scorecard• Proficiency targets for all schools
Michigan’s Accountability System
We believe:•Accountability only measures the core policy beliefs•The data in the accountability metrics is useful and necessary for schools and districts to understand their performance•Understanding where you are as a school/district is the first step toward moving forward•Working smarter, not just harder
Moving Beyond the Label
Three main components by subject:• Achievement• Improvement in achievement over time• The largest achievement gap between two subgroups calculated based on the top scoring 30% of students versus the bottom scoring 30% of students
Each component tells schools something about their overall performance and can be used for diagnostics
Using the Top to Bottom Ranking
Schools with 30+ full academic year (FAY) students over the last two years in at least two state-tested content areas; school must be OPEN at time of list generation
Application Some schools do not receive a ranking if they:
Have too few FAY students
Only have one year of data
Who receives a ranking?
Reading and Mathematics: Grades 3-8 and 11• In grades 3-8, testing every year allows us to figure out student performance level change (our current “growth” metric) in reading and math•Students can either significantly improve, improve, maintain, decline or significantly decline
Writing: Grades 4 & 7
Science: Grades 5 & 8
Social Studies: Grades 6 & 9
Tested Grades and Subjects
A school must change by four or more grades in order to get a new code • Example: A K-2 building becoming a K-6 building.• New codes NOT granted when a school is reopened as a charter, for example
If not, the school retains the old code and continues to have data “point” at it from all students for whom that code is their feeder schoolThere is no “phase reset” like there was in AYP• If school population changed by 51%, could request a phase reset—still got AYP calculations, but sanctions delayed• Under Priority/Focus interventions, would simply have a customized intervention.
What about Reconfigured schools?
Quick Reference for Z-Scores
Z-scores are a standardized measure that helps you compare individual student (or school) data to the state average data (average scores across populations).
Z-scores allow us to “level the playing field” across grade levels and subjects
Each Z-score corresponds to a value in a normal distribution. A Z-Score will describe how much a value deviates from the mean.
What do you need to know: Z-scores are used throughout the ranking to compare a school’s value on a certain component to the average value across all schools.
Why do We Use Z Scores?
Z-scores are centered around zero
Positive numbers mean the student or school is above the state average
Negative numbers mean the student or school is below the state average
What is a Z-Score?
0 1 2 3-1
-2
-3
State Average Better than state
average….…Worse than state
average
Your school has a z-score of 1.5. You are better than the state average.
Z-Score Examples
0 1 2 3-1
-2
-3
State Average Better than state
average….…Worse than state
average
Z-score of 1.5
Your school has a z-score of .2. You are better than the state average, but not by a lot.
Z-Score Examples
0 1 2 3-1
-2
-3
State Average Better than state
average….…Worse than state
average
Z-score of 1.5
Z-score of 0.2
Your school has a z-score of -2.0. You are very far below state average.
Z-Score Examples
0 1 2 3-1
-2
-3
State Average Better than state
average….…Worse than state
average
Z-score of 1.5
Z-score of 0.2
Z-score of -2.0
How Is the Top to Bottom Ranking Calculated
Two-Year Average
Standardized Student Scale
(Z) ScoreImprovement
Metric (Performance
Level Change OR Four Year
Improvement Slope)
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School Achievement
Z-Score
School Improvement
Z-Score
School Achievement Gap Z-Score
School Content
Area Index
1/2
1/4
1/4
Content
Index Z-
score
How Is the Top to Bottom Ranking Calculated
Two-Year Average
Standardized Student Scale
(Z) ScoreImprovement
Metric (Performance
Level Change OR Four Year
Improvement Slope)
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School Achievement
Z-Score
School Improvement
Z-Score
School Achievement Gap Z-Score
School Content
Area Index
1/2
1/4
1/4
Content
Index Z-
score
IMPORTANT PART RIGHT HERE!!!
Step #1: Take each student’s score on the test they took and compare that score to the statewide average for students who took that same test in the same grade and year
This creates a student-level z-score for each student in each content area• Compare• MEAP to MEAP• MEAP-Access to MEAP-Access• MME to MME• MI-Access• Participation to Participation• Supported Independence to Supported Independence• Functional Independence to Functional Independence
How do we get Standardized Scale Scores for Each Student?
Step #2: Once each student has a z-score for each content area (based on the test they took), we take all of the students in a each school, and rank order the students within the school.• Z-scores will have come from different tests, and compare students to statewide average for that grade, test, and subject• But they can now be combined for the school
Step #3: Add up all z-scores and take the average. This is now the average standardized student scale score.
Step #4: Define the top and bottom 30% subgroups, based on that rank ordering.
What do we do with those standardized scores?
Student Test Taken Z-score
Tommy Mi-Access, Participation
2.5
Sally MEAP 2.0
Maura MI-Access, SI 1.9
Fred MEAP 1.5
Ichabod MEAP-Access 1.0
Freud MEAP 0.8
Maybelle MI-Access, FI 0.7
Destiny MEAP 0.5
Harold MEAP -0.2
Bickford MI-Access, FI -0.5
Talledaga MEAP-Access -0.7
Francine MEAP -1.2
Joey MEAP -1.9
William MEAP -2.2
Student Test Taken Z-score
Tommy Mi-Access, Participation
2.5
Sally MEAP 2.0
Maura MI-Access, SI 1.9
Fred MEAP 1.5
Ichabod MEAP-Access 1.0
Freud MEAP 0.8
Maybelle MI-Access, FI 0.7
Destiny MEAP 0.5
Harold MEAP -0.2
Bickford MI-Access, FI -0.5
Talledaga MEAP-Access -0.7
Francine MEAP -1.2
Joey MEAP -1.9
William MEAP -2.2
Average Z-score (average standardized student
scale score): 0.28(sum all z-scores, divide
by 15)
Student Test Taken Z-score
Tommy Mi-Access, Participation 2.5
Sally MEAP 2.0
Maura MI-Access, SI 1.9
Fred MEAP 1.5
Ichabod MEAP-Access 1.0
Freud MEAP 0.8
Maybelle MI-Access, FI 0.7
Destiny MEAP 0.5
Harold MEAP -0.2
Bickford MI-Access, FI -0.5
Talledaga MEAP-Access -0.7
Francine MEAP -1.2
Joey MEAP -1.9
William MEAP -2.2
Top 30%
Bottom 30%
What is important to show Schools?
Two-Year Average
Standardized Student Scale
(Z) Score
Two-Year Average
Performance Level Change
Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School Achievement
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Content
Area Index
1/2
1/4
1/4
Content
Index Z-
scoreStep #1: AchievementHow well did the school do in that subject?
Positive number = better than averageNear zero = average
Negative number = worse than average
For grade 3-8 reading and mathematics
What is important to show Schools?
Two-Year Average
Standardized Student Scale
(Z) Score
Improvement Score
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School Achievement
Z-Score
School Improvement
Z-Score
School Achievement Gap Z-Score
School Content
Area Index
1/2
1/4
1/4
Content
Index Z-
score
Step #2: ImprovementIs the school improving in that subject?
Positive number = greater rate of improvement than average
Near zero = average improvementNegative = slower rate of
improvement than average; can also mean they are declining
For grade 3-8 reading and mathematics
What is important to show Schools?
Two-Year Average
Standardized Student Scale
(Z) Score
Improvement Value
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School Achievement
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Content
Area Index
1/2
1/4
1/4
Content
Index Z-
score
Step #3: Achievement GapIs the gap in that subject between
top 30% and bottom 30%:(positive number) = smaller gap
than average(negative number) = larger gap
than average(near zero) = average gap
What’s the overall pattern?• Low achievement? •Declining achievement?• Large gaps?
Where are the actionable areas?•Which subjects need the most attention?• Is everyone doing poorly (small gap, low achievement) or are some students doing well and others falling behind (decent achievement, but large gap)
Once they have looked at each component, Discuss:
For grade 3-8 reading and mathematics
What is important to show Schools?
Two-Year Average
Standardized Student Scale
(Z) Score
Two-Year Average
Performance Level Change
Index
Two-Year Average Bottom 30% - Top 30%
Z-Score Gap
School Achievement
Z-Score
School Performance Level Change
Z-Score
School Achievement Gap Z-Score
School Content
Area Index
1/2
1/4
1/4
Content
Index Z-
score
Focus Schools
Reward Schools (for
improvement)
My daughter’s elementary school
K-4 building
Go here to get this tool:•www.mi.gov/ttb
An example from data:
Accountability data does not tell ALL
“Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’.”
Moving beyond the label only takes capacity at every level of the field
Final Point
www.mi.gov/ttb• Individual school lookup tool•Diagnostic Worksheet
www.mi.gov/priorityschools
www.mi.gov/focusschools
www.mi.gov/rewardschools
Resources
