venessa a. keesler, ph.d. bureau of assessment and accountability michigan department of education...

Venessa A. Keesler, Ph.D.

Bureau of Assessment and Accountability

Michigan Department of Education

Presentation to MASFPS Fall Directors’ Institute

October 4, 2012

#1: NO ONE (and I do mean no one) likes accountability.

#2: If you are the one who is holding people accountable, you are not going to be the most popular person in the room

#3: See point number 1

Things I’ve learned about accountability since taking this job

Three myths; one reality•Myth #1: To drive reform•Myth #2: To create education policy•Myth #3: Because we are gluttons for punishment

Reality: • Accountability metrics/systems are quantitative articulations of the core policy beliefs of the education system• They help us measure our progress in meeting those core policy goals• They are the measure, not the purpose or the goal

So why do we do accountability?

A new era of accountability •Switching from a purely criterion-based system to a normative system•Criterion-based systems: Set average proficiency targets for schools.•Normative system: identifies the “worst” or “best” or “lowest” or “highest”

Accountability Landscape: 2012

Policy imperative for NCLB: all students CAN and SHOULD demonstrate proficiency criterion-system with proficiency targets for all schools and subgroups

10 years later: our average achievement is increasing, but we still have students and schools lagging behind

New policy imperative (ESEA Flex): we must target our lowest performing schools AND our lowest performing students more specifically and strategically

Why the change?

“Never trust an average”

Averages mask low performance

Example: • Proficiency target is 50%• Sunshine School has 20 students, 10 of whom are proficient and 10 of whom are not.• Sunshine School meets it’s target; hooray!• BUT those 10 not proficient students are possibly left behind

The problem with average proficiency rates at a school or district level

Top to Bottom Ranking• Priority Schools (bottom 5%)• Focus Schools (largest achievement gaps)• Reward Schools (high performing, high progress, beating the odds)

Accountability Scorecard• Proficiency targets for all schools

Michigan’s Accountability System

We believe:•Accountability only measures the core policy beliefs•The data in the accountability metrics is useful and necessary for schools and districts to understand their performance•Understanding where you are as a school/district is the first step toward moving forward•Working smarter, not just harder

Moving Beyond the Label

Three main components by subject:• Achievement• Improvement in achievement over time• The largest achievement gap between two subgroups calculated based on the top scoring 30% of students versus the bottom scoring 30% of students

Each component tells schools something about their overall performance and can be used for diagnostics

Using the Top to Bottom Ranking

Schools with 30+ full academic year (FAY) students over the last two years in at least two state-tested content areas; school must be OPEN at time of list generation

Application Some schools do not receive a ranking if they:

Have too few FAY students

Only have one year of data

Who receives a ranking?

Reading and Mathematics: Grades 3-8 and 11• In grades 3-8, testing every year allows us to figure out student performance level change (our current “growth” metric) in reading and math•Students can either significantly improve, improve, maintain, decline or significantly decline

Writing: Grades 4 & 7

Science: Grades 5 & 8

Social Studies: Grades 6 & 9

Tested Grades and Subjects

A school must change by four or more grades in order to get a new code • Example: A K-2 building becoming a K-6 building.• New codes NOT granted when a school is reopened as a charter, for example

If not, the school retains the old code and continues to have data “point” at it from all students for whom that code is their feeder schoolThere is no “phase reset” like there was in AYP• If school population changed by 51%, could request a phase reset—still got AYP calculations, but sanctions delayed• Under Priority/Focus interventions, would simply have a customized intervention.

What about Reconfigured schools?

Quick Reference for Z-Scores

Z-scores are a standardized measure that helps you compare individual student (or school) data to the state average data (average scores across populations).

Z-scores allow us to “level the playing field” across grade levels and subjects

Each Z-score corresponds to a value in a normal distribution. A Z-Score will describe how much a value deviates from the mean.

What do you need to know: Z-scores are used throughout the ranking to compare a school’s value on a certain component to the average value across all schools.

Why do We Use Z Scores?

Z-scores are centered around zero

Positive numbers mean the student or school is above the state average

Negative numbers mean the student or school is below the state average

What is a Z-Score?

0 1 2 3-1

-2

-3

State Average Better than state

average….…Worse than state

average

Your school has a z-score of 1.5. You are better than the state average.

Z-Score Examples

0 1 2 3-1

-2

-3

State Average Better than state

average….…Worse than state

average

Z-score of 1.5

Your school has a z-score of .2. You are better than the state average, but not by a lot.

Z-Score Examples

0 1 2 3-1

-2

-3

State Average Better than state

average….…Worse than state

average

Z-score of 1.5

Z-score of 0.2

Your school has a z-score of -2.0. You are very far below state average.

Z-Score Examples

0 1 2 3-1

-2

-3

State Average Better than state

average….…Worse than state

average

Z-score of 1.5

Z-score of 0.2

Z-score of -2.0

How Is the Top to Bottom Ranking Calculated

Two-Year Average

Standardized Student Scale

(Z) ScoreImprovement

Metric (Performance

Level Change OR Four Year

Improvement Slope)

Two-Year Average Bottom 30% - Top 30%

Z-Score Gap

School Achievement

Z-Score

School Improvement

Z-Score

School Achievement Gap Z-Score

School Content

Area Index

1/2

1/4

1/4

Content

Index Z-

score

How Is the Top to Bottom Ranking Calculated

Two-Year Average

Standardized Student Scale

(Z) ScoreImprovement

Metric (Performance

Level Change OR Four Year

Improvement Slope)

Two-Year Average Bottom 30% - Top 30%

Z-Score Gap

School Achievement

Z-Score

School Improvement

Z-Score

School Achievement Gap Z-Score

School Content

Area Index

1/2

1/4

1/4

Content

Index Z-

score

IMPORTANT PART RIGHT HERE!!!

Step #1: Take each student’s score on the test they took and compare that score to the statewide average for students who took that same test in the same grade and year

This creates a student-level z-score for each student in each content area• Compare• MEAP to MEAP• MEAP-Access to MEAP-Access• MME to MME• MI-Access• Participation to Participation• Supported Independence to Supported Independence• Functional Independence to Functional Independence

How do we get Standardized Scale Scores for Each Student?

Step #2: Once each student has a z-score for each content area (based on the test they took), we take all of the students in a each school, and rank order the students within the school.• Z-scores will have come from different tests, and compare students to statewide average for that grade, test, and subject• But they can now be combined for the school

Step #3: Add up all z-scores and take the average. This is now the average standardized student scale score.

Step #4: Define the top and bottom 30% subgroups, based on that rank ordering.

What do we do with those standardized scores?

Student Test Taken Z-score

Tommy Mi-Access, Participation

2.5

Sally MEAP 2.0

Maura MI-Access, SI 1.9

Fred MEAP 1.5

Ichabod MEAP-Access 1.0

Freud MEAP 0.8

Maybelle MI-Access, FI 0.7

Destiny MEAP 0.5

Harold MEAP -0.2

Bickford MI-Access, FI -0.5

Talledaga MEAP-Access -0.7

Francine MEAP -1.2

Joey MEAP -1.9

William MEAP -2.2

Student Test Taken Z-score

Tommy Mi-Access, Participation

2.5

Sally MEAP 2.0

Maura MI-Access, SI 1.9

Fred MEAP 1.5

Ichabod MEAP-Access 1.0

Freud MEAP 0.8

Maybelle MI-Access, FI 0.7

Destiny MEAP 0.5

Harold MEAP -0.2

Bickford MI-Access, FI -0.5

Talledaga MEAP-Access -0.7

Francine MEAP -1.2

Joey MEAP -1.9

William MEAP -2.2

Average Z-score (average standardized student

scale score): 0.28(sum all z-scores, divide

by 15)

Student Test Taken Z-score

Tommy Mi-Access, Participation 2.5

Sally MEAP 2.0

Maura MI-Access, SI 1.9

Fred MEAP 1.5

Ichabod MEAP-Access 1.0

Freud MEAP 0.8

Maybelle MI-Access, FI 0.7

Destiny MEAP 0.5

Harold MEAP -0.2

Bickford MI-Access, FI -0.5

Talledaga MEAP-Access -0.7

Francine MEAP -1.2

Joey MEAP -1.9

William MEAP -2.2

Top 30%

Bottom 30%

What is important to show Schools?

Two-Year Average

Standardized Student Scale

(Z) Score

Two-Year Average

Performance Level Change

Index

Two-Year Average Bottom 30% - Top 30%

Z-Score Gap

School Achievement

Z-Score

School Performance Level Change

Z-Score

School Achievement Gap Z-Score

School Content

Area Index

1/2

1/4

1/4

Content

Index Z-

scoreStep #1: AchievementHow well did the school do in that subject?

Positive number = better than averageNear zero = average

Negative number = worse than average

For grade 3-8 reading and mathematics

What is important to show Schools?

Two-Year Average

Standardized Student Scale

(Z) Score

Improvement Score

Two-Year Average Bottom 30% - Top 30%

Z-Score Gap

School Achievement

Z-Score

School Improvement

Z-Score

School Achievement Gap Z-Score

School Content

Area Index

1/2

1/4

1/4

Content

Index Z-

score

Step #2: ImprovementIs the school improving in that subject?

Positive number = greater rate of improvement than average

Near zero = average improvementNegative = slower rate of

improvement than average; can also mean they are declining

For grade 3-8 reading and mathematics

What is important to show Schools?

Two-Year Average

Standardized Student Scale

(Z) Score

Improvement Value

Two-Year Average Bottom 30% - Top 30%

Z-Score Gap

School Achievement

Z-Score

School Performance Level Change

Z-Score

School Achievement Gap Z-Score

School Content

Area Index

1/2

1/4

1/4

Content

Index Z-

score

Step #3: Achievement GapIs the gap in that subject between

top 30% and bottom 30%:(positive number) = smaller gap

than average(negative number) = larger gap

than average(near zero) = average gap

What’s the overall pattern?• Low achievement? •Declining achievement?• Large gaps?

Where are the actionable areas?•Which subjects need the most attention?• Is everyone doing poorly (small gap, low achievement) or are some students doing well and others falling behind (decent achievement, but large gap)

Once they have looked at each component, Discuss:

For grade 3-8 reading and mathematics

What is important to show Schools?

Two-Year Average

Standardized Student Scale

(Z) Score

Two-Year Average

Performance Level Change

Index

Two-Year Average Bottom 30% - Top 30%

Z-Score Gap

School Achievement

Z-Score

School Performance Level Change

Z-Score

School Achievement Gap Z-Score

School Content

Area Index

1/2

1/4

1/4

Content

Index Z-

score

Focus Schools

Reward Schools (for

improvement)

My daughter’s elementary school

K-4 building

Go here to get this tool:•www.mi.gov/ttb

An example from data:

Accountability data does not tell ALL

“Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’.”

Moving beyond the label only takes capacity at every level of the field

Final Point

www.mi.gov/ttb• Individual school lookup tool•Diagnostic Worksheet

www.mi.gov/priorityschools

www.mi.gov/focusschools

www.mi.gov/rewardschools

Resources

Venessa A. Keesler

keeslerv@michigan.gov (personal email)

mde-accountability@michigan.gov

877-560-8378, option 6

Contact Information