Child Outcome Data

Broad Data Analysis

Broad Analysis: Child Outcomes

Does our state’s data look different than the national data?

Are our state child outcomes trends stable over time?Is the data trending upwards?Is the data trending downwards?

Is our state performing more poorly in some outcomes than others?

Are the child outcomes similar across programs?

What about data quality? Can we be confident in our data?

Child Outcomes: State vs. National

Child Outcomes: State vs. National

National Vs. State Meaningful Differences

Child Outcomes: National vs. State FFY11 and State FFY12

Data Quality – Trends

Virginia Trends

Virginia Trends

Outcome 3:Use of Appropriate Behaviors to Meet

Needs

Outcome 2: Acquisition and

Use of Knowledge and Skills

Virginia Trends

Child Outcome Results FFY11/FFY12

Meaningful Differences Calculation

Child Outcomes: National vs. State FFY11 and State FFY12

Child Outcomes: Local vs. State

Child Outcomes: Local vs. State

Child Outcomes: Local vs. State

Child Outcomes: Local vs. State

Child Outcomes: Local vs. State

Child Outcomes: Local vs. State

Data Quality ElementsCompleteness of data

number of children reported for the outcome/number who exited

Virginia’s results: average= 65%; range for Local Systems = 17% - 100%

Expected Patterns for Progress Categories

Virginia’s state date is within these parameters for all three outcomes

Child Outcomes State Trends Over TimeAs noted on previous slides, Virginia’s results do not show

wide variations which would trigger concerns about data quality

Category a Category e

0 <5%

>10% >65%

Family Outcome Data

Broad Data Analysis

Broad Analysis: Family OutcomesDoes our state’s data look different than the

national data?Are our state family outcomes trends stable

over time?Is the data trending upwards?Is the data trending downwards?

Is our state performing more poorly in some outcomes than others?

Are the family outcomes similar across programs?

What about data quality? Can we be confident in our data?

Family Outcomes: State Trends over Time

Family Outcomes: Local vs. State 2012-20134A: EI has helped the family know their rights

Family Outcomes: Local vs. State 2012-20134B: EI has helped the family communicate their children’s needs

Family Outcomes: Local vs. State 2012-20134C: EI has helped the family help their child develop and learn

Data Quality

The data analysis for Virginia’ survey is quite extensive, using rigorous data analysis standards.

Virginia’s response rate, like other states that use mailing as the means to disseminate the family survey, is lower than states who use other mechanisms.

Broad Infrastructure Analysis

Analysis MechanismsUse of federal monitoring tools and

proceduresCommunication with Local Systems through:

Ongoing TARegional MeetingsTrainingsSystem Manager Meetings

Monitoring ResultsRecord Reviews

QMRsIndividualized TA

Local System Contract Deliverables

Infrastructure Analysis:Information Sources

Local self-reportingStakeholder inputLSM surveysTraining/meeting evaluationsMonitoring/QMRDispute ResolutionsObservations through TA and

other interactions

Broad Infrastructure Analysis

Governance - MixedStrengths

Code of Virginia establishes infrastructureVICCState Interagency Agreement; interagency

partnershipsDBHDS local contract with Local Lead Agencies

specifies LS infrastructure requirementsWeaknesses

Variable job roles and skill sets of LSMsVariable strength and specificity of LLA/provider

contracts Variable support from LLA leadership

Broad Infrastructure AnalysisFiscal - Mixed

StrengthsAdditional state fundingMedicaid EI Services programTrainings on fiscal issuesStrong working relationship with fiscal office at the

state lead agency (DBHDS)Weaknesses

Variable fiscal skill set of LSMsCode of Virginia does not require fiscal commitment at

local levelVariety of Local Lead Agencies Inconsistent reporting of fiscal data at local levelLack of fiscal data in state data systemPerception/reality that there is not enough money

Broad Infrastructure Analysis

Quality Standards – Not sure Practice Manual articulates expected

practices, but these are not labeled quality practices

Monitoring and Accountability – StrengthLocal self-monitoring and supervision for

continuous improvement – variableProfessional Development – StrengthTechnical Assistance – StrengthData – Weakness

Closing Thoughts

Based on this broad review, did anything

strike you as an area of focus for our Systemic

Improvement Plan?