final summary evaluation: state projects serving individuals with deaf-blindness october 23, 2006...

Final Summary Evaluation: State Projects Serving Individuals

with Deaf-Blindness

October 23, 2006

NCDB Sponsored Webinar

Presented by Richard Zeller

Presentation Overview

Self-assessment and verification review purpose and design

Summary of Year One (37) and Year Two (11) project self-assessments

Summary of on-site reviews

Feedback from projects and review teams

Evaluator Recommendations

Discussion

Evaluation Requirement

During year 2, each project must...

"conduct a comprehensive self-evaluation. The evaluation must include a review of the degree to which the project is meeting proposed objectives and goals and an evaluation of outcome data... In addition, the Department of Education intends to conduct a limited number of on-site evaluations based on a stratified randomized sample of sites.”

(RFP, page C-4)

Evaluation Purposes

Summative project evaluation questions: Are projects goals and objectives being achieved? Do projects address each RFP priority? Do projects have appropriate outcome data?

Formative project evaluation: Provide a continuous improvement process for individual projects to use.

National report - both summative and formative: provide information OSEP can use to guide needed system improvements.

Evaluation Constraints

48 projects - single and multi-state Staffing from partial to several FTE Common design to allow summaries Resources come primarily from the projects Assess whether projects are addressing RFP

priorities

Evaluation Design “Work scope” standards: Priorities (a) - (i) and

General Requirements (a) - (c) Priority questions:

What types of strategies are used? Is work being completed in a timely fashion? Are intended results being achieved? Are outcome data available (efforts and effects)? Are improvement plans in effect?

“General Requirements:” Are these requirements being appropriately addressed?

Evaluation Design (continued)

Self-assessments parallel MSIP’s Continuous Improvement and Focused Monitoring System (now the SPP & APR).

Verification Reviews (site visits) during this evaluation, reviews became a check on the self-evaluation process and a way to provide TA to the project.

Adjustments made in both self-assessment and review designs during implementation.

Self-Assessment Summary

Priorities (a) - (e): Strategies Timeliness Results Data - Effort and Effect Adjustments/Future Plans

Priorities (g) - (h) and General (a) - (c): Are the priorities addressed (Yes/No) Are there standards that apply?

Strategies Described

While the relative application of “ongoing” strategies was higher in year two, they

were also more distributed across projects than in year one.

Year One Year Two

Linear 1% 1%

Cyclical 15% 17%

Ongoing 36% 45%

Combined 48% 38%

For all timeliness item ratings

Year One

Year Two

active/behind schedule 15% 19%

not implemented/on schedule

1.4% 3%

active/on schedule 83% 78%

For all result item ratings:

Year One

Year Two

exceeding expectations 13% 13%

meeting expectations 72% 65%

below & approaching expectations

12% 20%

well below expectations 1% 1%

cannot rate 3% 1%

For all effort item ratings:

Ratings cluster: 3 projects in year one and 2 in year two rated “extensive” data for more than 6 items

Year One

Year Two

no data 2% 2%

some level of effort data 13% 33%

some quality of effort data 6% 7%

some level & quality of effort data 62% 45%

extensive level & quality of effort data

18% 13%

For all effect item ratings:Year One

Year Two

no data 12% 20%

outcome data 71% 68%

impact data 16% 12%

Clarification:• Outcomes are the immediate results of your

assistance (e.g., teacher skills gained in training)• Impacts are results your clients have when they

apply what you have taught them (e.g., they teach & children learn communications skills)

Overall Item Ranking (hi to lo)Year Two

(a)(3) R-B practices

(a)(1) State capacity

(d) Collaboration

(a)(4) Provider skills

(b)(1) Census

(a)(5) Address child/family needs

(e) Disseminate

(b)(2) Assess critical child needs

(a)(2) Systemic change

(c)(1) Evaluate effectiveness

(b)(3) Assess state needs

(c)(3) Advisory evaluation design

(c)(2) Measure child outcomes

Year One

(c)(3) Advisory evaluation design

(d) Collaboration

(b)(1) Census

(a)(1) State capacity

(a)(3) R-B practices

(a)(4) Provider skills

(b)(3) Assess state needs

(a)(5) Address family/child needs

(e) Disseminate

(a)(2) Systemic change

(c)(1) Evaluate effectiveness

(b)(2) Assess critical child needs

(c)(2) Measure child outcomes

Areas Needing More Attention?

(c)(2) Measure child outcomes (a)(2) Systemic change(b)(3) Assess state needs(b)(2) Assess child needs(c)(1) Evaluate effectiveness

(c)(3) Advisory evaluation design?

Adjustments/Future Plans

All projects/all strategies: about 81% (Year One) v. 63% (Year Two) of strategies are to “continue as proposed”

In Year One, 8 projects accounted for 55% of planned changes

In Year Two, all projects plan some changes in strategy, with 7 adopting new strategies

The most common areas of adjustment across both years were priorities (c)(1), evaluation and (c)(2), measurement of child outcomes

Priorities (g) & (h)

Affirmative Response:Year One

Year Two

(g) OSEP Directed TA 27% 0%

(g) Web-based TA 92% 82%

(g) Community of Practice 92% 82%

(h) Advisory Standards 97% 100%

(h) Act on Advisory Recs 100% 100%

(h) Advisory Change 38% 36%

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

General Priorities (a) - (c)

General Priority AreaYear One

Year Two

(a) Employ people with disabilities? 59% 73%

(a) Try to employ people? 68% 73%

(a) Advance people with disabilities? 62% 36%

(a) Change employment practices? 35% 36%

(b) Involve people with disabilities? 100% 100%

(c) Does project have a website? 86% 91%

(c) Is web-site accessible? 81% 82%

(c) Planning website improvement? 95% 45%

How Projects Relate Priorities to Work Design (Part 1, Year 2)

Objs or Goals Priority Cites Priorities / Obj Objs / Priority

33 41 1.2 1.7

20 38 1.9 1.1

23 88 3.8 1.2

34 164 4.8 1.8

20 120 6.0 1.1

6 42 7.0 0.3

26 182 7.0 1.4

14 116 8.3 0.7

3 27 9.0 0.2

14 133 9.5 0.7

10 111 11.1 0.5

Total Priority Cites by 11 Projects (Year 2, Part 1)(a)(1) State capacity 117

(a)(2) Systemic change

76

(a)(3) R-B practices 96

(a)(4) Provider skills 87

(a)(5) Assess family & child needs

64

(a)(6) Other 17

(b)(1) Census 34

(b)(2) Assess critical child needs

60

(b)(3) Assess state needs

54

(c)(1) Evaluate effectiveness

84

(c)(2) Measure child outcomes

49

(c)(3) Advisory evaluation 42

(d) Collaboration 70

(e) Disseminate 80

(g) OSEP specified TA 26

(h) Maintain Advisory 23

(a) Employ Individuals with disabilities

7

(b) Involve Individuals 43

(c) Web site accessibility 33

Verification Visit Summary

Sites Visited (in order): Year One: IN, FL, NJ, NY, WA, MO, CO, MT, CA Year Two: KY, MI , NC, TN

Process: Team of 3 reviewers each rated their agreement with the Project’s ratings for each priority and offered comments on each priority.

Revisions to the process and report form were made during the first year (simplifications) and again before year two (in response to suggestions).

Site Review Participation# Staff # Stakeholders

IN 4 20FL 5 12NJ 5 8NY 5 19WA 4 14CO 4 9MT 4 10MO 5 7CA 8 12KY 6 5MI 5 13NC 2 12TN 3 5

Who were the reviewers and how many sites did they visit?

Reviewer Name Year One Year TwoZambone 1Sharpton 1 1Bove 2McLetchie 2 2Rafalowski Welch 2Syler 2 2Fankhauser 3 1Dalke 4 3Rachal 4 2Steveley 6 1

Did site reviews tend to validate project self-assessments?

Agreement with project: “any reviewer’s rating of each project item self assessment rating”

Agreed with Project Year One Year Two Strongly Agree 85% 60% Mostly Agree 10% 25% Somewhat Agree 3% 2% Somewhat Disagree 2% 1% Strongly Disagree <1% 0%

Were reviewer ratings (after discussion) reliable?

Agreement here was defined as “all reviewers rated the way on a given item”

Year One: 32 actual disagreements, or 96.9% complete agreement on all items

Year Two: 1 disagreement, or 99%+ agreement on all items

Only 5 sites where disagreements among reviewers occurred; most of them in one site

How did projects and reviewers view the value of these two processes?

Self-assessment and improvement planning is a necessary function for the system of state projects.

The past and current processes and forms are complex and redundant, given the way the work is organized.

Both processes (self-assessments and review visits) have value, but both need substantial redesign.

Projects’ View of the Value of the Self-Assessments

Project Ratings (# reporting): Year 1 Year

2

High value 9 3

Moderate value 18 7

Some value 9 1

More trouble than it was worth 1

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.Year Two

What Some Projects Liked

Prompted communication with state program sites Forced staff to consider value of work The forms forced project to focus and limit narrative Improvement over earlier self-evaluation processes Aligned proposal to RFP, so not hard to use Format was easy and more logical [than year 1] Separate narrative allowed project to show how

priorities were woven into goals & objectives

What Projects Didn’t Like Priorities, criteria & proposed work not aligned Evaluation rules were not in the RFP Accessibility problems with form Redundancy (e.g., attachments & narratives) Word functions don’t work in the template form Too many reports for one year Too much time - takes away from TA Form accessibility (couldn’t enlarge print?) Narrative, priorities & ratings in three documents Format - impossible to match priorities to activities

Review Team Recommendations to Sites:

Expand partnerships (B, C, 619, others) - others must do the work of system change

Family networking/support (parent-to-parent) Define/structure TA and intent - child change,

local capacity building, systems change Systematize data collection (census, needs,

efforts and effects on individuals/systems) Use evaluation for program improvement

Review Team Suggestions

Better self-assessment instructions Consolidate Progress Report & Project Evaluation Clarify evaluation standards in the RFP Cluster priorities (eliminate redundancies) Value of the review process is the TA provided Effort & effect need better definition Change forms: Neither Year 1 or 2 worked for all Align Priorities and evaluation model

In future evaluation & review processes:

Evaluator Recommendations

The next RFP should have 5 program priorities (e.g., skill development, system capacity/change, child census/performance, family services, dissemination R-B practices)

Combine self-assessment and reporting in a single system with prescribed indicator measures for each priority for all projects

For larger projects (>$500K) adopt standard 3+2 procedures

Discussion:Were Evaluation Purposes Achieved?

Summative project evaluation questions: Are projects goals and objectives being achieved? Do projects address each RFP priority? Do projects have appropriate outcome data?

Formative project evaluation: Provide a continuous improvement process for individual projects to use.

National report - both summative and formative: provide information OSEP can use to guide needed system improvements.