building a data- driven culture in nevada june 10, 2013 the performance indicators project is a...

Building a Data-Driven Culture in Nevada

June 10, 2013

The Performance Indicators Project is a collaboration of the California Department of Social Services and UC Berkeley, and is supported by CDSS , and the Stuart Foundation.

Summit Content Areas• Leading with Data: A Focus on Outcomes

– Review of basic terminology– Avoiding pitfalls and data abuse– Key concepts in performance measurement– Telling the NV Story: State and National Context

• Connecting Data to Practice: Defining the problems

• From Data to Action: Strategy Development and Implementation

• CQI Structure and Function in NV

BASIC TERMINOLOGY AND FORMULAS

Data Analytics 101

Basic Terminology

Descriptive Data• Point-in-time

• Trends

• Comparisons

data source: AFCARS

Basic TerminologyProcess Measures - familiar to staff, relevant at a

caseworker level, current

Outcome Measures - the “big picture” measure of system performance, especially when looked at longitudinally

Measures of Central TendencyMean: the average value for a range of data

Median: the value of the middle item when the data are arranged from smallest to largest

Mode: the value that occurs most frequently within the data

12 4 15 63 7 9 4 17 4 4 7 9 12 15 17 63

4.168

631715129744 Mean

5.102

129 Median

4 Mode

7= 9.7

= 9

Measures of Variability

Minimum: the smallest value within the data

Maximum: the largest value within the data

Range: the overall span of the data

4 Minimum

63 Maximum

59463 Range

4 4 7 9 12 15 17 63

Disaggregation

• One of the most powerful ways to work with data…• Disaggregation involves dismantling or separating out

groups within a population to better understand the dynamics and plan strategies for improvement

• Useful for identifying critical issues that were previously undetected

Aggregate Permanency OutcomesRace/Ethnicity

Age

Region/Circuit

Placement Type

Measuring Change

100

yr baseline

yr) baseline-yr(latest netchange

•How much has this measure changed over time?•What will our performance be next quarter if we increase or decrease by 10%

10% increase = baseline x 1.1

10% decrease = baseline x .90

COMMON DATA PITFALLSData Analytics 101

Common Pitfalls

• Small N – impact on rates and trends• Seasonal variation• Faulty comparisons – failing to consider

demographic and policy differences• Outlier impact on central tendency• Data integrity/Data entry (over or under

emphasized)• Missing or incomplete definitions• Data overload: lack of focus on and connection

to key outcomes

Common Pitfalls: Seasonal variation

Period 1 to 2:38.5% reduction

Period 2 to 5:41.7% increase

Period 1 to 7:10.2% reduction

Common Pitfalls:Small n (impact on rates and trends)

100% reduction! But…from 2 children in

care to 0 children in care

57% increase!But from 7 to 11 children in

care

Common Pitfalls and Graph Interpretation

Data source: UNC at Chapel Hill Jordan Institute for Families website. URL: http://ssw.unc.edu/ma/

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

# initial place Group Home 25 25 17 23 11 5 9 4 2 1 7

Total Entries 269 215 236 253 236 194 190 197 148 156 150

% Initial place Group Home 9% 12% 7% 9% 5% 3% 5% 2% 1% 1% 5%

0%

5%

10%

15%

20%

0

50

100

150

200

250

300

% o

f all

entr

ies

# of

chi

ldre

n

Guilford County: First Entries by Initial Placement Type

http://ssw.unc.edu/ma/

Avoiding data overload and Managing with data

One thing the modern computer age has given everyone is data. Lots and lots of data. There is a large leap, however, between having data and learning from it.

W. Gregory Mankiw

Professor of Economics, Harvard

New York Times, Sunday Business Section (Sept. 5 2010) p. 5

Manage with Data

Provides us the ability to:• Compare metrics with agency mission and

practice model• Connect to evidence-based practice and link

processes to desired outcomes• Strategize on what work needs to be done• Focus on end outcomes• Identify what needs attention• Tell the story

Manage with Data

• Pick the right measures for the job• Prioritize reports and measures in line with

agency values, mission, vision• Connect process measures to outcomes/practice

model• Move beyond compliance and “gotcha”• Make it fun!• Celebrate success and tell the story• Use your data to engage the community, create

urgency for action, maintain support

Pick the Right Measures for the Job

• Federal Accountability• Overall Performance Monitoring• Outside Auditing (i.e. consent decrees,

monitors)• Contractor or Provider Performance• Office, Supervisor, Worker Level

accountability

Compliance Performance Outcomes

Continuum

Prioritize Measures in line with Agency Values

• To keep children safe and at home• To improve a child or youth’s well-being• To facilitate a child or youth’s move to swift

& certain permanency

Performance is Guided by Your Values as an Agency: Missouri Key Outcomes

Key Data Reports: How are they all connected?

CFSR and PIP

SCRT

ROMAFCARSNCANDS

SEE ResultsCOA

Connecting the Dots

Management Report: Frequency

of Visits with Caretakers

Case Review Measure: Caregiver involvement in case

planning

Case Review Measure:

Individualized Services

Outcome Measure:

Timely Reunifications

Process Data: AccountabilityRelevant to workers and

supervisors

Intermediate OutcomesRelevant to workers,

supervisors, managers

Outcomes: “So What?”Reflect Key Priorities of

Leadership

CFSR Findings: Relationship of Well-Being to Permanency

Positive ratings on

• Services to children, parents, foster parents

• Involvement of parents in case planning

• Caseworker visits with children

• Caseworker visits with parents

Substantial

achievement on

• Timely achievement of permanency

• Preserving children’s connections while in foster care

supports . . .

Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm

Factors Associated with Timely Reunification, Guardianship, and Permanent Relative Placement

The strongest associations with timely permanency included:

Caseworker Visits with Parents

Child’s Visits with Parents and Siblings in Foster Care

Services to Children, Parents, & Foster Parents

Family/Child Involvement in Case Planning

ASFA Requirements Regarding Termination of Parental Rights

Placement Stability


Strongest Associations Between Visits and Other Indicators

Both Caseworker Visits with Parents and Caseworker Visits with Children were strongly associated with:

Risk of harm to children

Needs & Services for children, parents, foster parents

Child and parent involvement in case planning


Other Significant Associations Between Visits and Indicators

Caseworker Visits with Parents and Caseworker Visits with Children were also strongly associated with:

Services to protect children at home Safety Outcome 1 Safety Outcome 2 Timely permanency goals Timely reunification Child’s visits with parents and siblings Relative placements Meeting educational needs Meeting physical health needs Meeting mental health needs


Connecting process to outcomes

Use Data to Create Urgency for Action –Target improvements based on your own baseline

Regional variation should generate productive discussion about differences in: • Service array• Community differences in reporting and tolerance for “risk”• Differences across partner agencies, courts, juvenile justice, behavioral health etc…• Demographic risk factors and “case mix”• Case loads, turnover (staff and leadership), and training• A variety of other policy/practice differences

Grounded in good case practice

model principles

Develop presentation

skills

Understand & demystify data

Master qualitative &

quantitative tools

Recognize challenges

Celebrate good practice

Support positive change

Act as a local resource

Grow as managers &

leaders

Your Role as a Data Leader

Knowing when you’ve got it right

• No more “the data are wrong”• Folks own data, know it, act on it• Practice people know the data, data people know

the practice• Field pulls data, asks for reports, initiates actions

tied to the data• Constantly talking about data in a positive way

After a really busy day, the data manager comments:

“I think I liked it better when no one paid attention to the data”

Telling the Story: Key Child Welfare Indicators

So What’s the Story?Describe the issue with as much detail as possible, variation is key to hypothesis development.

• What’s happening right now for all kids?• Has it always been this way?• Is it true in all places, for all ages, for all

racial/ethnic groups?• Is this indicator correlated with any others?• Does it look the same for all types of cases, or in

places where practice is different?

interdependence between measures…

CounterbalancedIndicators ofSystemPerformance

PermanencyThroughReunification,Adoption, orGuardianship

ShorterLengthsOf Stay

StabilityOf Care

Rate of Referrals/Substantiated Referrals Home-Based

Services vs.Out-of-HomeCare

Maintain Positive AttachmentsTo Family, Friends, andNeighbors

Use of LeastRestrictiveForm of Care

Reentry to Care

the current placement system*(highly simplified)

*adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference

CHILD INa bunch of

stuff happens CHILD OUT

the foster care system

Trends in Out of Home Care

Data source: AFCARS

Nationwide, the number of children in out of home care is declining. In NV, both the entry rate (per 1,000 children in the population) and the in-care rate are higher than the national average.

FY07 FY08 FY09 FY10 FY11 FY120

1000

2000

3000

4000

5000

6000

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

NV Trends

# in care on 9/30 <18

# entries in FY

# exits in FY

NV In care rate <18

National In care rate <18

NV Entry rate

National entry rate

Safety – The Absence of Repeat Maltreatment

New Y

ork

Iowa

Mich

igan

Nebra

ska

Flor

ida

Ohio

Califo

rnia

Illino

is

Natio

nal A

vg

Mai

ne

Distric

t of C

olum

bia

New Je

rsey

Nevad

a

Kentu

cky

Loui

siana

Wes

t Virg

inia

Mon

tana

Arizon

a

South

Car

olin

a

Delaw

are

New H

amps

hire

Kansa

s

Puerto

Rico

North

Car

olin

a

Virgin

ia

Verm

ont

82

84

86

88

90

92

94

96

98

100

Of all children who were victims of substantiated or indicated abuse or neglect during the first 6 months of the reporting year, what percent did not experience another incident of substantiated or

indicated abuse or neglect within a 6-month period? (FY10

CHILDREN ENTERING CAREManaging with Data in Child Welfare

Children Entering Care: Nevada

Key Questions: Entries

• What is the entry rate – by age/race?• Are entries increasing/decreasing? for all

groups?• What strategies are in place/planned to

reduce entries (and re-entries) into care?

Possible reasons for county differences in entry rates:

• Service array – preventive and in home • Standard of evidence• Law enforcement removals• Demographic risk factors• A variety of other policy/practice differences

Substantial variation year to year is also common in counties with few removals/small populations

CHILDREN IN CAREPOINT IN TIME

Managing with Data in Child Welfare

Key Questions: Children in Care

• What groups of children are in care NOW• What types of placements?• How long have they been in care?• What is needed to move them to

permanency?

Placement Type (Ages 0-17)

Was

hingt

on

New M

exico

Kansa

s

Distric

t of C

olum

bia

New J

erse

yId

aho

Louis

iana

Illino

is

Califo

rnia

(AFCARS)

Miss

ouri

Puerto

Rico

Florida

New H

amps

hire

Arizon

a

New Y

ork

Delawar

e

Virgini

a

Tenne

ssee

Mas

sach

uset

ts

Kentu

cky

Arkan

sas

Iowa

South

Dak

ota

North

Dak

ota

Wes

t Virg

inia

Rhode

Islan

d

Colora

do0%

5%

10%

15%

20%

25%

30%

35%

40%

Of all the children (age 0-17yrs) in care on the last day of the FY, what percent were placed in a congregate care setting? (Group home, shelter care, or residential facility:

excludes detention, and hospitalization)

Placement Type (ages 0-10)

Kansa

s

Louis

iana

Mich

igan

Was

hingt

on

India

naUta

h

Wisc

onsin

Califo

rnia

Mar

yland

Wes

t Virg

inia

Idah

o

Georg

ia

New J

erse

y

Hawaii

Delawar

e

Colora

do

Wyo

ming

Miss

issipp

i

Nation

al

North

Dak

ota

Mas

sach

uset

ts

Florida

Arizon

a

Mon

tana

Texas

South

Dak

ota

South

Car

olina

0%

2%

4%

6%

8%

10%

12%

Of all the children (age 0-10yrs) in care on the last day of the FY, what percent were placed in a congregate care setting? (Group home, shelter care, or residential facil-

ity: excludes detention, and hospitalization)

OUTCOMES: EXITS AND LENGTH OF STAY

National and State Level

Key Questions: Permanency Outcomes

•What proportion of children entering care will eventually reunify?•How does this differ by age at removal?•What percent of children remain in care after 3 years?•Are there differences by age/race?•Is this trend changing over time?

entry cohort

s

exit cohort

s

pointin time

data

Know which view to use

the view matters…

January 1, 2012 December 31, 2012July 1, 2012

Source: Aron Shlonsky, University of Toronto (formerly at CSSR)

the view matters…

0 5 10 15 20 25 30 35 40

q3(75% exited)

q2(50% exited)

q1(25% exited)

Nevada: Length of Stay in Months 2011

(children in care 5 days or more)

2011 entriesN=2103

Jan 1, 2011(point-in-time)N=4135

2011 exitsN=3138

Months in Care

insufficient time elapsed to determine this estimate

entries, point in time and exits views…

22

39

23

8 7

0

5

10

15

20

25

30

35

40

45

<1 yr 1-5 yrs 6-11 yrs 12-14 yrs 15-17 yrs

%

Nevada: Age of Children in Foster Care, 2012

Entries


22

39

23

8 78

37

29

1214

0

5

10

15

20

25

30

35

40

45

<1 yr 1-5 yrs 6-11 yrs 12-14 yrs 15-17 yrs

%


Entries

Point in Time


22

39

23

8 78

37

29

1214

5

43

27

1113

0

5

10

15

20

25

30

35

40

45

<1 yr 1-5 yrs 6-11 yrs 12-14 yrs 15-17 yrs

%


Entries

Point in Time

Exits

Exit Cohort View… but what about those that remain in care?

2007 2008 2009 2010 2011 20120%

10%20%30%40%50%60%70%80%90%

100%

Proportion of Exits by Type: Statewide

Death of child

Transfer

Runaway

No response

Live with relatives

Guardianship

Emancipation

Adoption

Reunification

Timely Reunification (entry cohort)

Puerto

Rico

Delawar

e

Verm

ont

Alaska

Arizon

a

Conne

cticu

t

Main

e

Virgini

a

Kansa

s

Florida

Califo

rnia

Nation

al

North

Dak

ota

New Y

ork

South

Dak

ota

Penns

ylvan

ia

New H

amps

hire

Wes

t Virg

inia

Rhode

Islan

d

Louis

iana

Wisc

onsin

Mas

sach

uset

ts

Tenne

ssee

Hawaii

Colora

do

Minn

esot

a

Arkan

sas

0

10

20

30

40

50

60

70

80

Timely Reunification (FY11): Measure C1.3 Of all first entries who remain in care at least 8 days, what % reunify within 12 months?

%

Nationally, there has been almost no improvement in timely reunification

FY05 FY06 FY07 FY08 FY09 FY100%

10%

20%

30%

40%

50%

60%

39%41% 42% 41% 41%

41%

37%39%

37%

41% 42% 44%

Timely Reunification (C1.3 Entry Cohort)

National Median NV

Re-Entry after Reunification

Puerto

Rico

Virgini

a

Main

e

Mich

igan

Delawar

e

Oklaho

ma

Kansa

s

South

Car

olina

Georg

ia

Hawaii

New M

exico

Arkan

sas

Utah

Distric

t of C

olum

bia

Nation

al

Califo

rnia

Kentu

cky

Mon

tana

Wes

t Virg

inia

New Y

ork

Wyo

ming

Florida

South

Dak

ota

Iowa

Wisc

onsin

New H

amps

hire

Penns

ylvan

ia0

5

10

15

20

25

30

Re-Entry (FY11) Measure C1.4 of all the children reunified, what % re-enter care within 12 months?

Permanency for Longer Stayers

FY05 FY06 FY07 FY08 FY09 FY100%

10%

20%

30%

40%

50%

60%

26% 26% 26%28% 29% 31%

31% 33% 32% 33% 34%

40%

Nationally, exits to permanency among children already in care two years or more has been improving (C3.1)

Nat'l Median NV

Permanency for Longer Stayers

Puerto

Rico

Delawar

e

Illino

is

Califo

rnia

Alabam

a

South

Dak

ota

Virgini

a

Mas

sach

uset

ts

Texas

South

Car

olina

Rhode

Islan

d

Verm

ont

Wisc

onsin

Nation

al

Kentu

cky

North

Car

olina

Iowa

Main

e

India

na

Mich

igan

Hawaii

Penns

ylvan

ia

Tenne

ssee

Was

hingt

on

Alaska

Arizon

a

Wes

t Virg

inia

0%

10%

20%

30%

40%

50%

60%

Achieving Permanency for Longer Stayers (FY11) Measure C3.1: Of all children in care at least two years, what % achieve permanency within the

following year?

Connecting Data to Practice: Using the CQI Framework

Continuous Quality Improvement (CQI)…an ongoing process of identifying, describing, and analyzing strengths

and problems and then testing, implementing, learning from, and

revising solutions.

CQI Relies on…

• An organizational culture that is proactive and supports continuous learning.

• A strong foundation – the mission, vision, and values of the agency.

• The active inclusion and participation of staff at all levels of the agency, children, youth, families, and stakeholders throughout the process.

Key Principles

• Use data and information from multiple sources, qualitative and quantitative

• Data have a purpose: Identify trends and anomalies; find areas for improvement; tell stories about what is happening in practice and policy

• CQI must support staff to improve outcomes for families

Key Principles

• If it ain’t “broke”, it can probably still be “fixed”

• CQI goes beyond “compliance” to “quality”• Meaningful and active engagement of staff

at all levels, children, youth, families, and stakeholders

• CQI requires training, preparation, and consistent ongoing support

DEMYSTIFYING THE LOGIC MODEL

CQI Group Exercise 2

Observe• We’ve noted that:• Children are not exiting to permanency

quickly enough

Explain• And we believe it is because:• Case management and case consultation

has not been consistent

Strategy• So we plan to:• Improve training and supervision; ensure

practice is aligned with policy

Outcome

• Which will result in ENVISIONED OUTCOME:

• An increase in children exiting to permanency within three years

Developed by NY OCFS

…if he had one hour to save the world he would spend 55 minutes defining the problem and only 5 minutes finding the solution.

Before jumping right into solving a problem• Step back• Invest time and effort• Improve understanding

Source: http://litemind.com/problem-definition/ (accessed 6/3/11)

http://litemind.com/problem-definition/

We have noted that:We believe it is because:

So we plan to:

Which will result in:

Observe Explain Prescribe Outcome

HYPOTHESIS STATEMENT: A HIGH LEVEL CAUSE AND EFFECT STATEMENT

Needs and Strengths Assessment Activities

Initial andIntermediate

Observe and Explain Strategies Outcome

LOGIC MODEL: DIGGING DEEPER – MORE DETAIL

OutputsKey End

Outcomes

Where are we now? Observe performance on key measures: review trends and patterns Establish priorities by considering: mandates, greatest areas of

need/opportunity for impact etc… Explain/Explore key underlying factors: both internal and external Consider subpopulations: is performance different by age? Race?

Maltreatment type? Define strengths & areas needing improvement

Administrative Data is only one part of the assessment.


So we plan to:





Short Term Outcomes



OutputsLong Term Outcomes

Where do we want to be? What are the ultimate outcomes that we hope to achieve?

Reduce entries into care Improve likelihood and timeliness of a permanent exit Reduce re-entry Improve health, mental health and education indicators

Group Exercise!!

Observe Explain Strategy Outcome

Observe

• We examined the data and noted that:

• How are you doing on key outcomes? Are they going in the right direction? Is this true everywhere, and for all children? What other indicators are related to this outcome?

Explain• And we believe it is because:• Why? Start with brainstorming, then look to

a variety of existing data – where is there variation? What more do you need to know? How will you find out?

Developed by NY OCFS

Table Discussion 1 – Trends in Timely Permanency

Review regional data packets and CQI handout:

Focus on WHAT and WHY• Describe the trends in timely permanency and related

measures• Are these indicators

– Increasing?– Decreasing?– Staying about the same?

• What does this tell us?• What more do we need to know?

MOVING FROM DATA TO ACTION

CQI Group Exercise 2


So we plan to:





Short term outcomes




What will we do to address the issue? Strategies should align with the strengths and needs. What activities are supporting good performance? What are the barriers?

Consider Strategies: What do you control? Where do you need to partner or advocate? Training Programs/Services Policies/practices


So we plan to:





Short Term Outcomes




How do we know that the strategy was implemented as planned? What are our timeframes?Examples of outputs: Often a count (and percent) # of people trained # of clients served # of referrals # of meetings held


So we plan to:





Short Term Outcomes




How will we know we are heading in the right direction?Short Term outcomes can be expected to change quickly.Examples of measurable improvements: Improve diligent search and engagement Reduce time to adjudication and disposition Increase timely permanency hearings Improved family engagement in case planning

building a data- driven culture in nevada june 10, 2013 the performance indicators project is a...

Documents

data slide

data range

data disaggregation

data maximum

care slide

reduction slide

nv slide

formulas data analytics