Evaluating the State Systemic Improvement Plan (SSIP)

ECTA CenterNational Center for Systemic Improvement

DaSy

Evaluating the Implementation

• Built into the plan from the beginning• Based on your theory of action• Based on data that informed the plan

development• Formative data and summative data• Evidence to show progress

Evidence of Progress

• Activities occurred and intended outcomes of each activity accomplished

• Changes are occurring at the system, practice and individual level

For Each Activity…

• Did the activity occur?– If yes, what evidence exists that it occurred?– If not, why not?– What needs to be done next?

• Did the activity accomplish the intended outcomes?– If yes, what evidence exists that the intended

outcomes were accomplished?– If not, why not?– What needs to be done next?

If the Activity is an Output/Product e.g. Training Resources

• How will we know if the activity occurs?– Evidences:

• Training materials developed• Training materials used as intended

• How will we know if the activity accomplishes the intended outcomes?– Evidences:

• Data indicating number frequency of training• Data indicating trainer fidelity and competency

If the Activity is a Process e.g. Training for Practitioners

• How will we know if the activity occurs?– Evidences

• Data indicating number of trainings• Participation records

• How will we know if the activity accomplishes the intended outcomes?– Evidences

• Data indicating frequency of use by participants• Measure of competencies

For Each Level of the System of Support…

• What are the guiding evaluation questions?• What are our performance measures to know

we’ve been successful?• What are the data sources for the

performance measures?

Measuring Results

• Results questions(s):– Have student outcomes improved?– Are school climates more collaborative?

• Performance measure:– % of students proficient and advanced– Improved collaboration among building educators

• Data sources:– MAP building outcome data– Self assessments and observations

Measuring Practices

• Practice-level questions:– Are more teachers implementing effective practices?– Is the use of effective practices increasing

• Performance measures:– % of practitioners implementing effective practices– % of buildings with 80% or more practitioners

implementing at a high level– Frequency of effective practices use

• Data sources:– Self-assessments, observations – Teacher evaluation summary data

Measuring Local Capacity

• Local system-level questions:– How has the local infrastructure been improved to

support implementation of the effective practices?• Performance measure:

– Changes to local system (governance, finance, personnel—TA and PD, data systems, expectations, teachers feel supported

• Data sources:– Collaborative climate, effective practices and data use

are part of the educator evaluation process

Measuring State Capacity

• State system-level questions:– How has the infrastructure been improved to support

implementation of the SSIP?• Performance measure:

– Changes to state systems (governance, finance, personnel—TA and PD, data systems, quality improvement, focus

• Data sources:– Plan process data sources– Plan outcome data sources– Observations related to time and focus

CW Data Update

March 2015Year 3

CW Events Associated w/ CW Buildings 7/1/2014 - 2/24/2015 by Type

RPDC

1 RPDC

2 RPDC

3 RPDC

4 RPDC

5 RPDC

6 RPDC

7 RPDC

8 RPDC

9

Professional Development

129 78 33 37 34 21 55 36 29

Coaching 48 30 48 88 39 65 45 202 14

Planning/ Communication/ Collaboration

50 19 59 19 31 11 80 70 31

State Meeting/ Conferences

0 0 0 0 1 0 0 0 0

Technical Assistance

12 8 2 38 0 7 0 119 14

Training 92 32 65 22 33 74 25 53 69

Other 0 0 2 1 0 0 0 1 0

Total 331 167 209 205 138 178 205 481 157

CW Events Associated w/ CW Buildings 7/1/2014 - 2/24/2015 by Type

RPDC

1 RPDC

2 RPDC

3 RPDC

4 RPDC

5 RPDC

6 RPDC

7 RPDC

8 RPDC

9

Professional Development

39.0% 46.7% 15.8% 18.0% 24.6% 11.8% 26.8% 7.5% 18.5%

Coaching 14.5% 18.0% 23.0% 42.9% 28.3% 36.5% 22.0% 42.0% 8.9%

Planning/ Communication/ Collaboration

15.1% 11.4% 28.2% 9.3% 22.5% 6.2% 39.0% 14.6% 19.7%

State Meeting/ Conferences

0.0% 0.0% 0.0% 0.0% 0.7% 0.0% 0.0% 0.0% 0.0%

Technical Assistance

3.6% 4.8% 1.0% 18.5% 0.0% 3.9% 0.0% 24.7% 8.9%

Training 27.8% 19.2% 31.1% 10.7% 23.9% 41.6% 12.2% 11.0% 43.9%

Other 0.0% 0.0% 1.0% 0.5% 0.0% 0.0% 0.0% 0.2% 0.0%

Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

CW Hours Associated w/ CW Buildings 7/1/2014 - 2/24/2015 by Type

RPDC

1 RPDC

2 RPDC

3 RPDC

4 RPDC

5 RPDC

6 RPDC

7 RPDC

8 RPDC

9 Planning/ Communication/ Collaboration

474 439 519 449 394 326 757 1732 725

Training 1012 382 648 585 512 533 292 602 247

Meetings 0 0 0 0 12 0 44 0 0

Travel 205 131 218 338 155 228 147 400 116

CW Events Associated w/ CW Buildings 7/1/2014 - 2/24/2015 by Topic

RPDC

1 RPDC

2 RPDC

3 RPDC

4 RPDC

5 RPDC

6 RPDC

7 RPDC

8 RPDC

9

Assessment-Capable Learners

28 39 34 15 14 39 35 28 2

Collaborative Data Teams 48 3 95 8 64 8 1 109 23

Common Formative Assessment

52 27 1 28 20 40 21 45 60

Collaborative Work Overview

91 14 16 28 6 21 25 111 22

Data-Based Decision Making 23 39 7 57 11 33 0 95 9

Effective Teaching/Learning Practices

19 0 1 16 1 6 43 1 2

Feedback 18 10 12 33 4 5 30 2 5 Metacognition 0 0 3 0 0 2 5 1 8 Reciprocal Teaching 27 23 33 20 15 24 36 78 13 Spaced vs Massed Practice

14 12 7 0 0 0 9 11 8

Other 11 0 0 0 3 0 0 0 5

CW Events Associated w/ CW Buildings 7/1/2014 - 2/24/2015 by Topic

RPDC

1 RPDC

2 RPDC

3 RPDC

4 RPDC

5 RPDC

6 RPDC

7 RPDC

8 RPDC

9

Assessment-Capable Learners

8.8% 23.4% 16.3% 7.3% 10.4% 21.9% 17.1% 5.8% 1.3%

Collaborative Data Teams 15.0% 1.8% 45.5% 3.9% 47.4% 4.5% 0.5% 22.7% 15.1%

Common Formative Assessment

16.3% 16.2% 0.5% 13.7% 14.8% 22.5% 10.2% 9.4% 39.5%

Collaborative Work Overview

28.4% 8.4% 7.7% 13.7% 4.4% 11.8% 12.2% 23.1% 14.5%

Data-Based Decision Making 7.2% 23.4% 3.3% 27.8% 8.1% 18.5% 0.0% 19.8% 5.9%

Effective Teaching/Learning Practices

5.9% 0.0% 0.5% 7.8% 0.7% 3.4% 21.0% 0.2% 1.3%

Feedback 5.6% 6.0% 5.7% 16.1% 3.0% 2.8% 14.6% 0.4% 3.3% Metacognition 0.0% 0.0% 1.4% 0.0% 0.0% 1.1% 2.4% 0.2% 5.3% Reciprocal Teaching 4.4% 7.2% 3.3% 0.0% 0.0% 0.0% 4.4% 2.3% 5.3% Spaced vs. Massed Practice

8.4% 13.8% 15.8% 9.8% 11.1% 13.5% 17.6% 16.2% 8.6%

Other 3.4% 0.0% 0.0% 0.0% 2.2% 0.0% 0.0% 0.0% 3.3%

Effective Teaching Practice of CFA by RPDC

RPDC 1 RPDC 2 RPDC 3 RPDC 4 RPDC 5 RPDC 6 RPDC 7 RPDC 8 RPDC 9 Unselected 10 2 2 0 0 0 3 15 9

Assessment Capable Learners 173 72 223 35 217 252 52 228 0 Classroom Discussion 0 10 1 0 0 0 0 2 0 Cooperative Learning 0 0 1 0 0 0 0 75 0

Differentiated Instruction 1 2 13 3 0 0 1 233 0 Direct Instruction 0 54 19 5 7 6 2 201 0

Feedback 94 43 152 196 0 110 152 380 237 Increasing Student Engagement 0 0 1 0 12 0 3 25 0

Metacognition 0 0 0 0 0 0 0 2 0 Reciprocal Teaching 15 8 1 0 0 31 2 48 2

Spaced vs. Massed Practice 30 3 13 0 16 122 0 13 0 Teacher-Student Relationship 0 0 0 0 0 0 0 4 0

Total 323 194 426 239 252 521 215 1226 248

Effective Teaching Practice of CFA by RPDC

RPDC 1 RPDC 2 RPDC 3 RPDC 4 RPDC 5 RPDC 6 RPDC 7 RPDC 8 RPDC 9 Unselected 3.1% 1.0% 0.5% 0.0% 0.0% 0.0% 1.4% 1.2% 3.6%

Assessment Capable Learners 53.6% 37.1% 52.3% 14.6% 86.1% 48.4% 24.2% 18.6% 0.0% Classroom Discussion 0.0% 5.2% 0.2% 0.0% 0.0% 0.0% 0.0% 0.2% 0.0% Cooperative Learning 0.0% 0.0% 0.2% 0.0% 0.0% 0.0% 0.0% 6.1% 0.0%

Differentiated Instruction 0.3% 1.0% 3.1% 1.3% 0.0% 0.0% 0.5% 19.0% 0.0% Direct Instruction 0.0% 27.8% 4.5% 2.1% 2.8% 1.2% 0.9% 16.4% 0.0%

Feedback 29.1% 22.2% 35.7% 82.0% 0.0% 21.1% 70.7% 31.0% 95.6% Increasing Student Engagement 0.0% 0.0% 0.2% 0.0% 4.8% 0.0% 1.4% 2.0% 0.0%

Metacognition 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.2% 0.0% Reciprocal Teaching 4.6% 4.1% 0.2% 0.0% 0.0% 6.0% 0.9% 3.9% 0.8%

Spaced vs. Massed Practice 9.3% 1.5% 3.1% 0.0% 6.3% 23.4% 0.0% 1.1% 0.0% Teacher-Student Relationship 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.3% 0.0%

Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

CFA Results for All Students by RPDC

All St

uden

ts

RPDC 1 RPDC 2 RPDC 3 RPDC 4 RPDC 5 RPDC 6 RPDC 7 RPDC 8 RPDC 9

Mid

-Instr

uction

Proficient and Higher 15767 5258 12490 2879 3508 18768 3453 22623 6134 Close to Proficient 8114 3790 9840 2056 1198 9908 1814 20931 4849

Far to Go 6455 2946 10294 1755 1984 7625 2561 26468 3893 Intervention Students 8188 2950 14159 1961 1746 6262 3565 30978 5579

Total 38524 14944 46783 8651 8436 42585 11393 101003 19092

After

Retea

ch Proficient and Higher 23732 7757 24907 5309 4723 26920 6050 50235 10128

Close to Proficient 7198 3781 10505 1671 1111 8691 1863 23361 4897 Far to Go 3862 1891 6387 766 1503 3939 1780 15847 2810

Intervention Students 3449 1525 5138 656 1096 2323 1494 10932 2856 Total 38241 14954 46937 8402 8433 41746 11187 100375 20691

Differe

nce Proficient and Higher 7965 2499 12417 2430 1215 8152 2597 27612 3994

Close to Proficient -916 -9 665 -385 -87 -1217 49 2430 48 Far to Go -2593 -1055 -3907 -989 -481 -3686 -781 -10621 -1083

Intervention Students -4739 -1425 -9021 -1305 -650 -3939 -2071 -20046 -2723 Total -283 10 154 -249 -3 -839 -206 -628 1599

CFA Results for All Students by RPDC

All St

uden

ts

RPDC 1 RPDC 2 RPDC 3 RPDC 4 RPDC 5 RPDC 6 RPDC 7 RPDC 8 RPDC 9

Mid

-Instr

uction

Proficient and Higher 40.9% 35.2% 26.7% 33.3% 41.6% 44.1% 30.3% 22.4% 30.0% Close to Proficient 21.1% 25.4% 21.0% 23.8% 14.2% 23.3% 15.9% 20.7% 23.7%

Far to Go 16.8% 19.7% 22.0% 20.3% 23.5% 17.9% 22.5% 26.2% 19.0% Intervention Students 21.3% 19.7% 30.3% 22.7% 20.7% 14.7% 31.3% 30.7% 27.3%

Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

After

Retea

ch Proficient and Higher 62.1% 51.9% 53.1% 63.2% 56.0% 64.3% 54.1% 50.0% 48.9%

Close to Proficient 18.8% 25.3% 22.4% 19.9% 13.2% 20.8% 16.7% 23.3% 23.7% Far to Go 10.1% 12.6% 13.6% 9.1% 17.8% 9.4% 15.9% 15.8% 13.6%

Intervention Students 9.0% 10.2% 10.9% 7.8% 13.0% 5.5% 13.4% 10.9% 13.8% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

Differe

nce Proficient and Higher 21.1% 16.7% 26.4% 29.9% 14.4% 20.2% 23.8% 27.6% 19.0%

Close to Proficient -2.2% -0.1% 1.3% -3.9% -1.0% -2.5% 0.7% 2.5% 0.0% Far to Go -6.7% -7.1% -8.4% -11.2% -5.7% -8.5% -6.6% -10.4% -5.5%

Intervention Students -12.2% -9.5% -19.3% -14.9% -7.7% -9.2% -17.9% -19.8% -13.5%

CFA Results for Students with Disabilities by RPDC

SWDs

RPDC 1 RPDC 2 RPDC 3 RPDC 4 RPDC 5 RPDC 6 RPDC 7 RPDC 8 RPDC 9

Mid

-Instr

uctio

n Proficient and Higher 959 261 469 95 271 1322 116 1361 208

Close to Proficient 707 239 505 130 135 1298 71 1516 272 Far to Go 813 302 758 182 216 1603 210 2772 346

Intervention Students 1539 505 1723 335 381 1483 596 5023 1163 Total 4018 1321 3455 742 1003 5433 993 10672 2006

Not Assessed 367 24 124 49 47 492 106 917 111

After

Rete

ach

Proficient and Higher 1679 414 1071 281 370 2348 314 3359 430 Close to Proficient 810 295 795 195 124 1259 143 2340 387

Far to Go 609 288 765 125 211 903 197 2388 363 Intervention Students 857 278 872 130 299 815 330 2548 792

Total 3955 1305 3503 732 1004 5293 984 10604 2028 Not Assessed 416 57 120 43 44 531 106 884 89

Diffe

renc

e

Proficient and Higher 720 153 602 186 99 1026 198 1998 222 Close to Proficient 103 56 290 65 -11 -39 72 824 115

Far to Go -204 -14 7 -57 -5 -700 -13 -384 17 Intervention Students -682 -227 -851 -205 -82 -668 -266 -2475 -371

Total -63 -16 48 -10 1 -140 -9 -68 22 Not Assessed 49 33 -4 -6 -3 39 0 -33 -22

CFA Results for Students with Disabilities by RPDC

SWDs

RPDC 1 RPDC 2 RPDC 3 RPDC 4 RPDC 5 RPDC 6 RPDC 7 RPDC 8 RPDC 9

Mid

-Instr

uctio

n Proficient and Higher 23.9% 20.0% 13.6% 12.8% 27.0% 23.2% 11.7% 12.8% 10.5%

Close to Proficient 17.6% 18.3% 14.6% 17.5% 13.5% 22.7% 7.2% 14.2% 13.7% Far to Go 20.2% 23.1% 21.9% 24.5% 21.5% 28.1% 21.1% 26.0% 17.4%

Intervention Students 38.3% 38.6% 49.9% 45.1% 38.0% 26.0% 60.0% 47.1% 58.5% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

Not Assessed N/A N/A N/A N/A N/A N/A N/A N/A N/A

After

Ret

each

Proficient and Higher 42.5% 32.5% 30.6% 38.4% 36.9% 44.1% 31.9% 31.6% 21.8% Close to Proficient 20.5% 23.1% 22.7% 26.7% 12.4% 23.6% 14.5% 22.0% 19.6%

Far to Go 15.4% 22.6% 21.8% 17.1% 21.0% 17.0% 20.0% 22.5% 18.4% Intervention Students 21.7% 21.8% 24.9% 17.8% 29.8% 15.3% 33.5% 24.0% 40.2%

Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Not Assessed N/A N/A N/A N/A N/A N/A N/A N/A N/A

Diffe

renc

e

Proficient and Higher 18.6% 12.5% 17.0% 25.6% 9.8% 20.9% 20.2% 18.8% 11.3% Close to Proficient 2.9% 4.9% 8.1% 9.2% -1.1% 0.9% 7.4% 7.8% 5.9%

Far to Go -4.8% -0.5% -0.1% -7.4% -0.5% -11.1% -1.1% -3.5% 1.0% Intervention Students -16.6% -16.8% -25.0% -27.4% -8.2% -10.7% -26.5% -23.1% -18.3%

Total N/A N/A N/A N/A N/A N/A N/A N/A N/A Not Assessed N/A N/A N/A N/A N/A N/A N/A N/A N/A