the improvement of child protective services with structured ......the improvement of child...

The Improvement of Child Protective Services with Structured Decision Making

The CRC Model

Children’s Research CenterA Division of the National Council on Crime and Delinquency

The Improvement of Child Protective Services with Structured Decision Making

The CRC Model


“CPS units have been plagued by long-standing systemic weaknesses in day-to-

day operations, including difficulty in maintaining a skilled workforce;

consistently following key policies and procedures designed to protect

children; developing useful case data and recordkeeping systems, such as

automated case management; and establishing good working relationships with

the courts.”

Complex Challenges Require New Strategies,

United States General Accounting Office, July 1997


Children’s Research Center426 S. Yellowstone Dr., Suite 250

Madison, WI 53719608.831.1180

Fax 608.831.6446

NCCD Headquarters Office685 Market Street, Suite 620

San Francisco, CA 94105415.896.6223

Fax 415.896.5109

www.nccd-crc.org

Janice Ereth, Ph.D., Director

Dennis Wagner, Ph.D., Director of Research

S. Christopher Baird, Senior Vice President

Rod Caskey, Senior Program Manager

Terry McHoskey, ACSW, Senior Program Manager

Raelene Freitag, M.S.W., Ph.D., Senior Researcher

Susan Gramling, J.D., Senior Researcher

Peter Quigley, Senior Researcher

Rick Wiebush, Senior Researcher

Deb Paulus, Office Manager

© Copyright 1999

All Rights Reserved

NCCDNATIONAL COUNCIL ON CRIME AND DELINQUENCY

TABLE OF CONTENTS

LIST OF FIGURES

PREFACE

THE STATE OF CHILD WELFARE SERVICES ......................................................................................................1

THE STRUCTURED DECISION MAKING MODEL: A NEW APPROACH TO DECISION MAKING AND CASE MANAGEMENT ......................................................................................3

Principles ............................................................................................................................................................3

System Components............................................................................................................................................4

The Structured Decision Making Tools ............................................................................................................6Responding to Allegations of Abuse/Neglect..............................................................................................7Assessing the Threat of Imminent Harm....................................................................................................9Assessing Future Risk ................................................................................................................................11

Risk Classification and Re-Referral Rates ..........................................................................................13Assessment of Family Needs and Strengths..............................................................................................15Reassessment of Risk and Needs ..............................................................................................................15Establishing Service Standards for Cases at Each Risk Level ..................................................................17Structured Decision Making for Children in Out-of-Home Care ..........................................................18

The Management Components of the SDM Model ......................................................................................20Workload Measurement ............................................................................................................................20Management Information Reports: Data for Planning, Monitoring and Evaluation ................................21

DOES THE SDM MODEL WORK? ......................................................................................................................24

The OCAN Study ..............................................................................................................................................24Risk Assessment Reliability ........................................................................................................................24Risk Assessment Validity ............................................................................................................................25

Evaluation of the Michigan SDM System........................................................................................................26

CONCLUSION ........................................................................................................................................................29

REFERENCES ..........................................................................................................................................................30

APPENDIX A

Michigan Assessment for Substantiated Cases, Family Risk Assessment of Abuse/Neglect ........................a1Rhode Island Family Classification for Risk of Abuse/Neglect ......................................................................a2Rhode Island Substantiation Rates by Risk Levels ..........................................................................................a3New Mexico Risk Assessment Study, Outcomes by Risk Level Assigned ......................................................a3

APPENDIX B

California Family Strengths and Needs Assessment (Child Section) ............................................................b1

LIST OF FIGURES

Figure 1: Children’s Research Center Structured Decision Making Systems ......................................2

Figure 2: Objectives of the Structured Decision Making System Model ............................................4

Figure 3: Summary of Structured Decision Making System Components ........................................5

Figure 4: Minnesota Response Priority for Physical Abuse ..................................................................8

Figure 5: Georgia Department of Human Resources Safety Assessment and Plan ..........................10

Figure 6: California Family Risk Assessment ......................................................................................12

Figure 7: California Risk Assessment Outcomes by Risk Level ..........................................................13

Figure 8: Wisconsin Urban Caucus Family Strengths and Needs Assessment ..................................16

Figure 9: Michigan CPS Service Standards..........................................................................................17

Figure 10: Placement/Permanency Plan Guidelines..............................................................................19

Figure 11: Determination of Workload Demand and Staff Needed ....................................................21

Figure 12: Changes in Initial Risk Levels ..............................................................................................22

Figure 13: Substance Abuse Rehabilitation Needs, Referrals and Outcomes ......................................22

Figure 14: Mean Number of New Investigations and SubstantiationsReported During an 18-Month Follow-Up Period by Risk Level ..........................................25

Figure 15: Michigan SDM Evaluation Results:Percent of High Risk CPS Cases that Received Specific Services............................................27

Figure 16: Michigan SDM Evaluation Results:Outcomes for CPS Cases, 12-Month Follow-Up ................................................................28

The Children’s Research Center (CRC) wasestablished to help federal, state and local childwelfare agencies reduce child abuse and neglect bydeveloping case management systems andconducting research which improves servicedelivery to children and families. The Center is adivision of the National Council on Crime andDelinquency (NCCD), which was established in1907 to perform a similar role for private andpublic agencies serving delinquent children.NCCD, which employs staff at offices in SanFrancisco, California, and Madison, Wisconsin, isone of the oldest nonprofit research and advocacyagencies in the United States. During the last twodecades, the agency has conducted research,evaluated programs and developed case manage-ment systems for more than 50 state or federalagencies. In 1986, after successful completion of arisk assessment model for Alaska’s delinquentpopulation, that state’s Social Services agencyasked NCCD to work with Child ProtectiveServices (CPS) staff to devise a system that wouldprovide the same level of structure for CPS. Thisinitial project began NCCD’s expansion ofstructured decision making (SDM) principles andpractices to the child protection field.

The success of the Alaska CPS project led tosimilar efforts in the early 1990’s in Michigan,Oklahoma, Rhode Island, and Wisconsin. Duringthe past five years, NCCD’s work in child welfarehas increased dramatically. Indiana, Georgia, andNew Mexico have all designed and implementedstructured decision making models. So too has oneof Australia’s six states (South Australia).California, Minnesota and Ohio beginimplementation in 1999. Colorado will

implement some aspects of the system in 1999 aswell. With each new project, we have honed ourunderstanding of the needs of child welfareagencies and what is required to successfullyimplement major organizational change. Inaddition, we have assembled a substantialresearch database and developed systems formonitoring service delivery, improving efficiency,and measuring the effectiveness of child welfarepolicies, programs, and services.

Improving child protection systems has been aformal part of NCCD’s mission since 1993, whenour Board of Directors authorized the creation ofa special division called the Children’s ResearchCenter (CRC). Many abused and neglectedchildren later become involved in delinquent andcriminal behavior, ending up in substance abuseprograms, training schools, jails, and prisonsthroughout the nation. To stem the cycle of crimeand violence in the United States, organizationslike NCCD must focus on improving services tofamilies and children. The CRC mission is tocontinue research and evaluation efforts in childwelfare and to assist agencies to improve theirservice delivery systems. Meeting the needs of at-risk children and families will create a better, safersociety for all Americans.

This document outlines CRC’s approach to riskassessment and structured decision making inchild welfare. Examples of research results,decision support systems, and data we havecompiled are shown throughout this booklet toillustrate the value of the SDM model. We believeyou will find the materials informative andthought provoking. For additional information,please contact the Children’s Research Center.

PREFACE

Janice Ereth, Ph.D., Director, Children’s Research Center

Chris Baird, Senior Vice President, National Council on Crime and Delinquency

The Structured Decision Making Model 1

The number of abuse and neglect allegationsnationwide has risen dramatically over the lasttwo decades. Most child welfare agencies havebeen hard pressed to respond effectively, as thenew demands have outpaced availableresources. The results have included lawsuits inmore than 30 states, media exposes resultingfrom child deaths, increased concerns overworker and agency liability, and a continuoussearch for new strategies and resources toaddress the burgeoning problem.

The need for additional resources is obvious,but that is not the only issue. The increasingpressures have highlighted a problem that haslong plagued human services agencies ingeneral, and child welfare agencies in particular:the need for more efficient, consistent and validdecision making. Child protection workers areasked to make extremely difficult decisions, yetin many agencies, workers have widely differentlevels of training and experience. Consequently,decisions regarding case openings, childremoval and reunification, and other service-related issues have long been criticized asinappropriate, inconsistent, or both. In fact,research has clearly demonstrated that decisionsregarding the safety of children varysignificantly from worker to worker, evenamong those considered to be child welfareexperts (Rossi, et.al.,1996). As pressure tomake critical decisionsaffecting children andfamilies rises, so does thepotential for error.Inappropriate decisions can be costly, leading toan overuse of out-of-home placements, ortragic, resulting in the injury or death of a child.

The problems of increasing referrals, limitedresources, and liability concerns are inextricablylinked with decision making issues. Agenciesoverwhelmed by heavy workloads need to beable to consistently and accurately determinewhich cases should be investigated, whichchildren need to be removed, and which familiesrequire the most intensive services. Clearly, newmethods are required to help agencies and

workers make decisions as efficiently andeffectively as possible. Tools are needed thathelp workers make accurate and reliableassessments of immediate safety issues andlonger-term risk. Decision making strategies areneeded that help focus limited resources onthose families at higher levels of risk. Thesedecision tools must be embedded in casemanagement systems that incorporate clearlydefined service standards, mechanisms forfrequent reassessments, methods for measuringworkload, and mechanisms for ensuring

accountability and qualitycontrols.

How child welfaredecisions get made andhow agency resources areutilized are the key issues

addressed in the CRC structured decisionmaking model. While the model does notpurport to be a “cure-all” for the current crisisin child welfare, that crisis cannot be overcomeuntil the issues surrounding child welfaredecision making are confronted. We believe themodel described in this document - whenproperly implemented - will result in asignificant step forward for child welfareservices. This model is based on workcompleted or underway in fourteen states,ranging from California to Rhode Island.

THE STATE OF CHILD WELFARE SERVICES

Inconsistent and inappropriate decisions are the heart of the

problem facing America’s child welfare agencies.

The State of Child Welfare Services

The State of Child Welfare Services2

Risk Scale Development and/or SDM Implementation

Children's Research CenterStructured Decision Making Systems

Note: SDM has also been implemented in South Australia.

Figure 1

THE STRUCTURED DECISION MAKING MODEL:A NEW APPROACH TO DECISION MAKING AND CASE MANAGEMENT

Principles

The structured decision making modeldescribed in this document is based on fourprimary principles. First, decisions can besignificantly improved when structuredappropriately: that is, specific criteria must beconsidered for every case by every workerthrough highly structured assessmentprocedures. Failure to clearly define decisionmaking criteria and identify how workers are toapply these criteria results in inconsistenciesand, sometimes, inappropriate case actions.

The second principle stipulates that prioritiesgiven cases must correspond directly to theresults of the assessment process. Expectationsof staff must be clearly defined and practicestandards must be readily measurable. Theassessment process has little meaning unlessresults lead directly to an indicated decision.And those decisions should be structured sothat the highest agency priority is given to themost serious and/or highest risk cases.Moreover, if prioritization is to be translatedinto practice, there must be clearly identified -and implemented - differential servicestandards associated with each type of case.Service standards, differentiated by level of risk,provide a level of accountability that is oftenmissing in human service organizations.

The third principle is that virtually everythingan agency does - from providing services to anindividual case to budgeting for treatmentresources - is a response to the assessmentprocess. For example, risk and needs assessmentsshould be directly linked to service plans. In theaggregate, assessment data also will helpindicate the range and extent of serviceresources needed in a community. Similarly,

assessment and case classification results aretied directly to agency service standards, whichin turn drive staff workload and budgetingissues.

Fourth, a single, rigidly defined model cannotmeet the needs of every agency. All state andcounty child welfare agencies are not organizedto deliver services in the same way and do notalways share similar service mandates. As aresult, the CRC approach to system develop-ment is a collaborative one in which agenciesare engaged in a joint development effort. Eachsystem is built upon a set of principles andcomponents which are then adapted to localpractices and mandates, incorporating a greatdeal of input from local managers and staff. Theresult is a site-specific system which is “owned”by the agency and builds upon its strengths as aservice organization.

The priority assigned to casesmust correspond directly to the

results of a clear, concise and carefully structured assessment process.


System Components

The SDM model hasseveral basic com-ponents. At the heartof the system is a seriesof tools used to assessfamilies and structurethe agency response tothem. The followingtools are used atmultiple decisionpoints, ranging fromintake to reunification:

• a response prioritydecision system toguide how quickly

investigative staff should respond to areferral alleging child abuse/neglect;

• a safety assessment to determine the threatof immediate harm and identify stepsneeded to protect children;

• a research-based risk assessment to estimatethe likelihood of future abuse/neglect;

• standardized assessments of family andchild strengths and needs to guide serviceplanning; and

• periodic reassessments of safety, risk, andneeds to determine the need for changes inservice levels and/or changes inpermanency planning.

A second basic component is the use ofservice levels (e.g., low, medium, high) withminimum standards for each level. Theservice levels and associated standards aredesigned to ensure that staff time andattention is concentrated on those families atthe highest levels of risk and need.

Finally, the model also includes twomanagement-related components:

• A workload measurement and accountingsystem for determining the number of childwelfare staff needed to meet the workloaddemand, and for distributing workloadequitably among staff.

• A management information componentthat uses aggregate family assessment data,agency response/decision data andworkload data to assist managers inplanning, monitoring, budgeting andevaluation.

4 The Structured Decision Making Model

OBJECTIVES OF THE STRUCTURED DECISION MAKING MODEL

• To introduce structure to critical decision points in the child welfare system.

• To increase the consistency and validity of decision making.

• To target resources to families most at risk.

• To improve the effectiveness of Child Protective Services.

Figure 2


Figure 3

SUMMARY OF STRUCTURED DECISION MAKING SYSTEM COMPONENTS

COMPONENT WHEN USED PURPOSE/DECISIONS METHOD OF DEVELOPMENT

1. Response Priority At time of referral Accept referral or not, Policy/consensushow quickly to respond

2. Safety Assessment At onset of investigation; prior to Identify immediate threat of Policy/consensus/researchany removal and when considering harm and potential protectingreturn interventions/removal

3. Risk Assessment By completion of investigation Assess long-term likelihood of Research - risk studyre-abuse or re-neglect, open or close decision, level of service

4. Family Strengths and By completion of investigation Assess family strengths/needs, Policy/consensusNeeds Assessment (typically for opened cases) help determine level of service,

drive case plan

5. Child Strengths and By completion of investigation Assess child’s strengths/needs, Policy/consensusNeeds Assessment (typically for those entering drive child’s service plan

out-of-home care)

6. Classification and At completion of risk/needs Differentiate levels of service for PolicyService Standards opened cases

7. Risk/Needs Reassessment Every 3-6 months Measure progress, adjust service Research/policylevel, amend case plan, case closure

8. Reunification Assessment When considering return Reassess risk, safety, compliance with Research/policyfrom foster care case plan and visitation

9. Workload Management Ongoing Assess number of staff needed, Research - workload studyworkload allocation, case assignments

10. Management Information Ongoing Monitor quality assurance, Aggregate data: assessmentplanning, evaluation, budgeting results; service referrals;

workload; outcomes

The Structured Decision Making Tools

The SDM system brings structure andconsistency to each decision point in the childwelfare system through the use of assessment

tools that are objective,comprehensive, and easy touse. The structured assess-ments ensure that eachfamily is systematicallyevaluated and that criticalcase characteristics are notoverlooked. They are in astraightforward, simpleformat that is seldom morethan one or two pages inlength. This allows criticalcase information to be docu-mented in a short time. Therelative ease of application isparticularly critical foragencies where staff turnoveris high, there are largenumbers of inexperiencedstaff, and/or workload is

threatening to overwhelm staff.

The model uses different tools for each decisionpoint because there are different issues thatneed to be addressed at each stage of the case.The issues associated with determiningresponse priority for example, are quitedifferent from those required to assess thenature of services needed for an open case. Nosingle instrument can successfully capture ororganize the disparate issues that must beconsidered at each distinct point of caseprocessing.

The assessment tools are not intended to makecase decisions for direct service workers. Staffstill need to exercise professional judgement.But the various tools help structure decisions bybringing objective information to bear on thesecritical questions:

• Are factors present that indicate the childmay be in immediate danger if left in thehome during the investigation?

• What is the likelihood that abuse or neglectwill recur in this family in the near future?

• What specific family issues need to beaddressed in order to reduce risk?

• What relative priority for agency serviceresources should this family receive?

• Can the child be returned home?

Staff in states that have implemented the SDMmodel generally believe that the assessmenttools help them focus on critical issues, andprovide a basis for explaining and justifyingtheir decisions. In short, staff view theassessment tools as mechanisms which helpthem work with families more effectively.


The model uses different criteria ateach decision point to address theissues at each stage of the case.

Responding to Allegations ofAbuse/Neglect

The initial call alleging abuse or neglecttypically requires staff to answer two questions:1) is this an allegation of abuse or neglect?; and,if so, 2) how quickly do we need to initiate theinvestigation? These “front door” questionshave major implications for child safety and foragency workload. Yet, all too often agency policyabout what should or should not be investigatedis vaguely defined or not clearly understood bystaff. Even when it is clear that the allegation isabuse/neglect related, the criteria fordetermining the urgency of the case and thespeed of the agency’s response often varies bythe unit, the supervisor, and/or the intakeworker involved.

The CRC SDM system clearly identifies factorsthat determine how quickly staff shouldrespond to new child abuse/neglect referrals.This results in greater consistency amongworkers and also permits administrators to easily convey to key decision makers and the

general public how the agency deals with abuseand neglect referrals.

An example of a response priority decisionsystem is shown in Figure 4. This “decision tree”approach incorporates and prioritizes criticalfactors and leads staff to a decision about the

speed of the response. (The figure shows thecriteria for responding to allegations of abuse.Separate decision trees are used for other typesof allegations.)


The questions of whether and how quickly to respond to an allegation have major implications for child

safety and agency liability.


Minnesota Response PriorityPhysical Abuse

Are significant bruises, contusions, or burns evident or is medical care required?

Is child under age seven orlimited by disability?

Were severe or bizarre methodsused or was abuse premeditated

Will perpetratorhave access to childin next 48 hours?

Will perpetrator have access to child in next 48 hours

or is child afraid to go home?

Have there been prior investigatedreports of abuse?

Are domesticviolence issues

present?

Is non-perpetratorcaregiver's response

appropriate andprotective of child?

Level 1

Level 1Level 1

Level 2

Level 2

Level 2

Level 2

Level 2

Level 3

yes no

yes noyes no

yes no

yes no

yes no

yes no

yes no

Level 1 requires a face-to-face response within 24 hours.Level 2 requires a face-to-face response within 3 working days.Level 3 requires a face-to-face response within 5 working days.

Figure 4


Assessing the Threat of Imminent Harm

Perhaps the most critical decision facing childwelfare workers is whether to leave an abused orneglected child in the home while services toreduce risk of harm are put into place. It is adifficult decision, with major implications forthe safety of children, their long-termpsychological development, family functioning,worker liability, and the professional image ofthe agency. Yet, as documented in a majornational study of child welfare decision making(Rossi, et.al., 1996), there is no consistentagreement among child welfare workers andexperts about the conditions that warrantremoval from the home. Sadly, one of the keystudy findings was that “a family’s chances ofhaving a child taken into custody varies widelyaccording to the person who is assigned toinvestigate that case” (Rossi, et. al., 1996).

To address this concern, the SDM modelincorporates a safety assessment protocol (seeFigure 5) that is adapted from a modeloriginally developed in New York (Salovitz,1993). The purpose of the tool is:

• to help workers assess whether and to whatextent any children are in immediate dangerof serious physical harm;

• to determine what interventionsshould be initiated or maintainedto provide appropriate protection;and

• if sufficient protection cannot beprovided, to establish criteria foremergency removal.

At the onset of an investigation, staff must beable to assess child safety concerns, and developand implement appropriate safety plans. Thesafety assessment facilitates these tasks byrequiring workers to: 1) focus attention on a setof 10-12 specified, and clearly defined,conditions that potentially represent a threat tochild safety; and 2) identify the interventionsneeded to control and remediate any unsafe

condition(s). Children are considered to be“unsafe” when any safety factor is present andthe only intervention considered sufficient toprotect them is removal. The safety assessmentis also completed when considering a returnhome for any children who had previously beenremoved.

The CRC model makes a clear distinctionbetween “safety” issues and “risk” issues. Safetyrefers to imminent danger to the child and thesteps required to provide immediate, short-term protection. Risk, on the other hand,represents the likelihood that the caretaker willre-abuse or re-neglect the child in the future.The concept and practice of risk assessment isdescribed in the following section.

The safety assessment helps workers focusattention on a set of 10-12 specified, andclearly defined, conditions that potentially

represent a threat to child safety.


Assessing Future Risk

The family risk assessment scales are research-based tools which estimate the likelihood that afamily will again become involved in an abuseor neglect incident. They are the result ofresearch that examines the relationship betweenfamily characteristics and child welfare caseoutcomes. The risk scales, which incorporate arange of family characteristics (e.g., number ofprior referrals, children’s ages, caretakersubstance abuse), have all demonstrated a

strong correlation with subsequent childabuse/neglect referrals. One very importantresearch finding is that a single instrumentshould not be used to assess the risk of bothabuse and neglect. Different family dynamicsare present in abuse and neglect situations.Hence, separate risk scales are used to assess thefuture probability of abuse or neglect. Figure 6shows the empirically-based abuse and neglectrisk assessment tools recently developed for theState of California.

Because these tools are products of research onthe actual experience of families previouslyreported to the agency, it is possible to assessrisk with a reasonably high degree of accuracy.Recent research has indicated that riskinstruments are transferrable amongjurisdictions (Baird, Wagner, Healy, andJohnson, 1999). However, when research is doneusing cases from the local jurisdiction theagency has added confidence that the riskinstrument reflects local conditions and thelocal child welfare population.

The risk assessment concept is simple. Thescales are used to classify families into riskgroups that have high, medium or lowprobabilities of continuing to abuse or neglect

their children. For instance, in many of the CRCrisk assessment studies, it often has beenpossible at the completion of the investigationto identify “high risk” families that have a 50%or higher probability of again abusing orneglecting their children. It has also beenpossible to identify “low risk” families where thechances of subsequent maltreatment were only5% or lower.

The differences between these groups aresubstantial. High risk families are far morelikely than low risk families to re-abuse their

children. The research has shown thathigh risk families have significantlyhigher rates of subsequent referralsand investigations, more subsequentsubstantiations, and are more ofteninvolved in serious abuse or neglectincidents resulting in medical careand/or hospitalization. Armed with

this critical information, agencies are well-positioned to make decisions about howresources should be differentially allocatedacross clients.

CRC risk assessment instruments havedemonstrated their ability to classify

families into risk groups that have high,medium or low probabilities of continuing

to abuse or neglect their children.

Risk Classification and Re-Referral Rates

Figure 7 illustrates results from a recent CRCstudy (California, 1998). A random sample of2,511 families referred to child welfare fromseven California counties was included in theresearch. As the figure demonstrates, there is astrong relationship between risk levels and alloutcome measures.

For example, the data show that among familiesclassified as low risk, just 18.5% had asubsequent referral for abuse or neglect duringthe 24-month follow-up period. In contrast,among families classified as very high risk, there-referral rate was 60%, more than three timesthe rate found for low risk cases.

Importantly, in all CRC studies to date, the risksystems developed also promote equity in

decision making. Because equity is a majorprinciple of the development process, theproportion of African Americans, Whites andHispanics assigned to each risk level is virtuallyidentical in all jurisdictions. These resultssuggest that well-structured assessment toolsand decision making systems can helpovercome some of the racial disparitiesresulting from traditional practices.


0%

10%

20%

30%

40%

50%

60%

70%

Low Moderate High Very High

18.5%

7.7%

1.4%4.3%

27.5%

13.8%

6.5% 7.0%

60.1%

44.3%

27.8%

22.7%

48.0%

31.6%

20.4%

14.4%18.5%

7.7%

1.4%4.3%

27.5%

13.8%

6.5% 7.0%

60.1%

44.3%

27.8%

22.7%

48.0%

31.6%

20.4%

14.4%

Subsequent Referral

Subsequent Placement

Subsequent Substantiation

Subsequent Injury

California Risk AssessmentOutcomes by Risk Level

Percent

Results of other risk studies are presented in Appendix A.

Figure 7

N=2,511

In many child welfare agencies, inexperiencedworkers, minimal training, and high turnoverall but guarantee that clinical judgments of riskwill vary widely among workers.Line staff often fail to identifyhigh risk families duringabuse/neglect investigationsand therefore do not engagethem in services (Johnson andL’Esperance, 1984). CRCresearch shows that in someagencies using traditional assessment methods,many (in some instances, most) high risk casesare not opened for services while manylow risk families are carried oncaseloads for years. The result is thatagencies are losing the opportunity toprevent abuse in the families who aremost at risk. By using actuarial riskassessment, child welfare agencies candirectly address this issue andsignificantly improve the initial caseservice decisions made by individualworkers.


Risk assessment can help line staff make better decisions. Research has consistently demonstrated

that simple actuarial tools can assess risk more accurately than even a well-trained clinical staff person (Meehl

1954; Dawes, Faust, and Meehl 1989).

Assessment of Family Needs and Strengths

Another important feature of the CRCstructured decision making system is the familystrengths and needs assessment. A companionpiece to the risk assessment, it is used tosystematically identifycritical family issuesand resources andhelp plan effectiveservice interventions.The strengths andneeds assessment instrument shown on thefollowing page was designed - using a consensusapproach - in collaboration with staff fromcounties in Wisconsin. This and similar toolsserve several purposes:

• It ensures that all workers consistentlyconsider each family’s strengths andweaknesses in an objective format whenassessing need for services.

• It provides an important case planningreference for workers and first line supervisorswhich eliminates long, disorganized casenarratives and reduces paperwork.

• It provides a basis for monitoring whetherappropriate service referrals are made.

• The initial needs assessment, when followedby periodic reassessments, permits caseworkers and supervisors to easilyassess change in family functioningand thus monitor the impact ofservices on the case.

• It provides management withaggregated information on theissues client families face. Theseprofiles can then be used todevelop resources to meet clientneeds.

Some jurisdictions have also developeda child-focused strengths and needsassessment tool for use with childrenwho may be placed out of the home.An example is shown in Appendix B.

Reassessment of Risk and Needs

The initial assessments of risk and service needsare followed by routine reassessments, which areconducted at established intervals (generallyevery 90 days) as long as the case is open. Case

reassessment ensuresthat any changes in riskor family service needswill be consideredin la ter s tages o fthe service delivery

process, and that case decisions will be madeaccordingly. Case progress will determine if alower or higher service level is needed, or ifthe case can be closed. In most agencies, therisk and needs assessment/reassessmentinstruments have become formal case planningdocuments and thus reduce the need for longcase narratives and other paperwork. The timesaved is available to actually serve families.

Periodic reassessment also provides for ongoingmonitoring of important case outcomes suchas: 1) new abuse or neglect incidents; 2) out-of-home placement status of children in thefamily; 3) changes in each family’s serviceutilization pattern; and 4) changes in theseverity of previously identified needs. In short,the reassessment of each family at fixedintervals provides direct service workers and

their supervisorswith an efficientmechanism forcollecting andevaluating infor-mation necessary toeffectively managetheir cases.


The strengths/needs assessment is used to systematically identify

family issues. It provides afoundation for the service plan.

Establishing Service Standards for Casesat Each Risk Level

Not all families involved in child abuse orneglect incidents require the same level of childwelfare services. Yet in terms of case assignmentand resource allocation, many child welfareagencies treat each case the same.Hence, services are sometimesprovided to families who will notbenefit from them, while otherhigher risk families do not receivethe resources needed to adequatelyprotect children.

Risk assessment provides an objectiveframework for making service decisions. Theability to more accurately assess risk allowsagencies to target service resources moreefficiently. A primary mechanism for targetingresources is the use of differential service

standards, whereby the mandated frequency ofcontact between the worker and the family istied to the family’s level of risk. Low riskfamilies need not receive the same amount ofagency resources (i.e., case worker time) as highrisk families because they are much less likely toagain maltreat their children. When differentialworker contact standards based on risk are

established by an agency, it becomes possible tomake existing service resources reach fartherand produce better results. Figure 9 shows howthe Michigan Family Service Agency has definedand differentiated service standards by casetype. Similar standards have been implementedin many other agencies.


MICHIGAN CPS SERVICE STANDARDS

Service Level Minimum Standards

Low 1 face-to-face contact by the CPS worker with client per month, plus

1 collateral contact per month by the worker on behalf of the client

Moderate 2 face-to-face contacts by the CPS worker with client per month, plus

2 collateral contacts per month by the worker on behalf of the client

High 3 face-to-face contacts by the CPS worker with client per month, plus


Intensive 4 face-to-face contacts by the CPS worker with client per month, plus


Linking service standards to risk assures thatresources are targeted to families most likely

to again abuse or neglect their children.

Figure 9

Structured Decision Making for Childrenin Out-of-Home Care

CRC has applied the principles of standardizedassessment and structured decision making tofamilies with children in foster care. The intentof the foster care component is to ensure thatstate and federal policies regardingreunification, permanency planning forchildren, and termination of parental rights areeffectively translated into practice. To this end,the model establishes presumptive guidelinesfor children in care based on: 1) risk of futuremaltreatment; 2) the safety of the homeenvironment; and 3) demonstrated parentalinterest and involvement in the lives of theirchildren. It is a “best practice” guide that willfacilitate implementation of the new federallegislation while leading to more consistentand appropriate decision making. While everyagency will need to modify this component ofthe model to include its own assessmentinstruments, policies, and terminology, theoverall logic of the system is universallyapplicable. The system presumes the following:

• When families reduce risk to an acceptablelevel and maintain appropriate visitationwith their children, the child should bereturned home if the home is judged to besafe.

• When risk remains high or the homeremains unsafe or parents fail to meet theirvisitation responsibilities for a specifiedperiod time (in concert with federalguidelines and agency policy), it ispresumed that the goal will be changedfrom return home to another plan forpermanency.

In the foster care model, the initial risk level isestablished using the research-based riskassessment instrument. The risk reassessmentassumes that risk is reduced if the family hasmade significant progress toward treatmentgoals. The reassessment scoring systemgenerally precludes consideration ofreunification if there had been any newsubstantiation of maltreatment of any child inthe household since the previous assessment.

The reunification model consists of fourassessment components:

• a structured risk reassessment;

• a structured evaluation of parentalcompliance with visitation schedules;

• a reunification safety assessment; and,

• structured guidelines for changing thepermanency planning goal.

As shown in Figure 10, the results of thestructured assessments (Risk, VisitationCompliance, Safety) are jointly considered toguide decisions regarding return to the home orchanges in the permanency plan. This ispresented as an example. In practice, CRC staffwork with each agency to develop a protocolthat incorporates criteria reflective of key localpolicies and regulations.



Placement/Permanency Plan GuidelinesIs Risk Level Low or Moderate?

Has risk remained high or very high for three consecutive assessments?*

*Agency policy would determine the number of assessments conducted before a change in the permanency plan is indicated.

Have parents maintained an acceptable levelof compliance with visitation plan?

Has the child been in placement12 consecutive months or

15 of last 22 months?

Has the home been rated unsafe for three consecutive

assessments?*

Is the home safe or safewith services in place?Maintain in

Out-of-HomeCare

ChangePermanency

Plan Goal

Maintain inOut-of-Home

Care

ChangePermanency

Plan Goal

Maintain inOut-of-Home

Care

ChangePermanency

Plan Goal

Return Home

no yes

no yes

no yes

no yes

no yes

no yes

Figure 10

THE MANAGEMENT COMPONENTSOF THE SDM MODEL

In addition to providing greater consistency indecision making and more efficient use ofresources, the SDM model includes twocomponents designed specifically to facilitatemanagement and administration of the childwelfare agency. These components - workloadmeasurement and management informationreports - build upon and help maximize theutility of the structured decision making aspectsof the model.

Workload Measurement

Workload measurement is based on theassumption that simple caseload counts do notadequately capture the amount of time - andtherefore the number of staff - needed to fulfill

the child welfareagency’s mandates.Moreover, giventhe delineation ofdistinct case typesand differentialservice standardsin the SDM model,caseload counts arean ineffectivemeasure for deter-mining how work-load should bedistributed acrosswork units orindividual staff.

Workload measurement translates “caseload”into time requirements and ultimately, staffingneeds. To establish a workload system, a simplecase-based time study is conducted todetermine the amount of time actually needed bystaff to meet service standards for various typesof cases. This information is used to calculatethe agency’s total “workload demand,” whichcan then be compared to the current “supply” of

available staff. Knowing the monthly timerequirement for each case type, and the totalworkload demand, allows the agency to:

• provide a rational, empirical basis forbudget and staffing requests to externalfunding sources;

• develop an internal system for equalizingworkload across staff or work units; and

• estimate the impact of new serviceresponsibilities or budget restrictions onagency service delivery.

Since the agency is able to specify its case-related service standards, and identify thenumber of staff required to serve cases

according to those standards, a workload-based budget in essence becomes a contractfor services. Funding bodies will know exactlywhat level of service will be provided basedon the level of staff resources allocated. Theeffect of budget reductions on client servicewill be readily apparent, as will the effect ofenhanced resources.

Figure 11 provides an example of a workload-based budget.


Workload measurement translates “caseload” into time requirements and

ultimately, staffing needs.


Management Information Reports: Datafor Planning, Monitoring and Evaluation

An important feature of the CRC model is thatit can provide management with information toroutinely monitor and evaluate programs,assess the impact of policy, identify serviceneeds and determine which programs andintervention strategies provide the best resultsfor various types of cases. A basic premiseunderlying CRC’s approach to managementinformation is that the information needed tomake good decisions at the individual case level(e.g., structured assessments of risk and serviceneeds) is the same information needed - inaggregate form - by agency supervisors, analystsand administrators.

As shown in Figure 12, aggregated riskinformation can, for example, documentchanges in the nature of the client population.This (example) graph reveals substantialincreases over a five-year period in theproportion of substantiated cases identified ashigh and very high risk. This informationclearly demonstrates new challenges facing theagency and documents changes in workload.Similarly, Figure 13 shows how managers canuse needs and service referral data to monitorthe extent to which clients are receiving servicesfor identified problems, and the effectiveness ofthose services upon subsequent substantiations.

Figure 11

DETERMINATION OF WORKLOAD DEMAND AND STAFF NEEDED (Example)

Time Required Time Required

Case Type Number of Cases Per Case, Per Month Per Month

Per Month to Meet Standards by Case Type

Abuse/Neglect Intakes 700 1.0 hrs 700 hrs

Investigations 350 6.5 hrs 2,275 hrs

Ongoing Cases

Low 100 3.3 hrs 330 hrs

Medium 250 4.2 hrs 1,050 hrs

High 150 6.6 hrs 990 hrs

Foster Care Cases 100 9.0 hrs 900 hrs

TOTAL WORKLOAD DEMAND 6,245 hrs/month

WORKER TIME AVAILABLE PER MONTH 1 120.6 hrs

TOTAL NUMBER OF STAFF NEEDED 2 51.8

1. Time available to handle cases. Reflects reductions from salaried hours due to vacation, sick days, training and administrative tasks.

2. Calculated by dividing workload demand by time available per worker.


74.8% 25.2%

Received Services48.2%

Services Unavailable1.7%

18.5%

No Referral37.4%

Services Refused12.7%

6.7%

Families with Serious/Chronic Problems

Referrals NewSubstantiations

Source: Michigan Family Independence Agency

Substance Abuse RehabilitationNeeds, Referrals, and Outcomes

0%

10%

20%

30%

40%

50%

60%

1993 1998

10%

50%

25%

15%

7%

43%

30%

20%

Low Moderate High Very High

Changes in Initial Risk Levels1993 - 1998

(Example)

Figure 12

Figure 13

SDM management information can also beused to increase the agency’s evaluationcapabilities. The organization can establishclearly defined outcome objectives for policiesand programs and use the aggregate datagenerated by the CRC model to determine theextent to which those objectives were realized. Aconsortium of counties in Wisconsin, forexample, using data routinely generated by the

CRC system, were able to: 1) revalidate their riskassessment instrument, and 2) demonstrate thatproviding intensive services to high and veryhigh risk cases significantly reducedsubsequent referrals for abuse and neglect.These data have profound implications forfuture funding and resource allocation.

In sum, the CRC approach: 1) provides theability to critically evaluate programs essentialfor improving services to families and children,and 2) directly enhances an agency’s evaluationcapacity by providing quality data on clientcharacteristics, system processing and caseoutcomes.


The data generated by the SDM system can be used to evaluate child

welfare policies and programs.

The SDM model incorporates laudable goals.The components of the model make conceptualand intuitive sense, and they are informed byextensive research. But a key question remains:does the SDM model work? That is:

• are case decisions in fact more consistentacross staff?

• does SDM help staff make better decisions?

• does the model actually have an impact interms of reducing the incidence ofsubsequent abuse and neglect?

The results of two evaluations that addressedthese questions are presented below.

The OCAN Study

A variety of risk assessment tools have beendeveloped and adopted by child welfareagencies. Until recently however, these riskmodels, whether consensus- or empirically-based, had not been rigorously evaluated. Toremedy this, the Office of Child Abuse andNeglect (OCAN) selected the Children’sResearch Center (overseen by an independentAdvisory Board of national experts) to conducta comparative evaluation of the reliability and

validity of three different risk models (Baird,Wagner, Healy and Johnson, 1999). Theseincluded two consensus models - theWashington Risk Assessment Matrix and theCalifornia Family Assessment Factor Analysis (aderivative of the Illinois CANTS model) - aswell as the empirically-based Michigan FamilyRisk Assessment.

Risk Assessment Reliability

The first phase of the study assessed thereliability of the three risk models bymeasuring the extent to which differentworkers assigned the same risk level to thesame family. The study methodology involveda total of 80 randomly selected cases whichwere assessed by four case readers who hadbeen trained in the Washington scale, fourothers who had been trained to use theCalifornia instrument and four others who hadbeen trained in the Michigan model.1 Bothsimple comparisons of the percentage of caseson which raters agreed and a statistical measureof reliability, Cohen’s Kappa, demonstrated thatthe reliability of the Michigan system wassignificantly higher than the level of reliabilityattained by the “expert” or “consensus-based”approaches to risk assessment.

24 Does the Structured Decision Making Model Work?

DOES THE SDM MODEL WORK?

The reliability of the empirically-basedMichigan model was significantly

greater than the consensus models tested in a national study.

1 The study took place in four different sites to ensure broad geographic and ethnic representation - Alameda County (Oakland), CA;Dade County (Miami), FL; Jackson County (Kansas City), MO; and four counties in Michigan (Macomb, Muskegon, Ottowa andWayne). The 80 sample cases consisted of 20 selected from each site. There were three people in each site - each trained on a differentrisk instrument - who completed the risk assessments. Each site team scored the 20 cases from its site and the cases from each of theother sites.

A significantly higher level of reliability wasfound for the Michigan system than either ofthe two consensus-based approaches. In 85% ofall cases, at least three of the four raters agreedon the risk level assigned to a case. Reliability ofthe consensus based instruments, however, waswell below what is considered adequate. Whenthe Washington or California systems were usedto rate risk, substantial differences among theraters were noted. For both systems, at leastthree of the four raters agreed on a risk level inonly about 50% of all cases assessed. A statisticaltest used to measure reliability (Cohen’s Kappa)indicated the Michigan scale was reliable, whilethe Washington and California scales were not.

Risk Assessment Validity

The second phase of the OCANstudy evaluated the validity ofthe California, Michigan andWashington risk assessmentsystems. “Validity” refers to theextent to which an instrumentin fact measures what it purportsto measure. In the child welfarerisk context, the fundamentalevaluation questions for assessingthe validity of risk instruments areas follows:

• Does a higher risk classificationindicate a greater probability of re-referral for abuse or neglect?

• Are there substantial differences in re-referral rates between risk classifications?Ideally, “high” risk cases should have a re-referral rate that is three to four times greaterthan the cases classified as “low” risk.

To assess the validity of the three riskinstruments, CRC compared results from acohort of 1,400 cases investigated for abuse andneglect allegations in the Fall of 1995. Followingthe investigations, each family was tracked for18 months. Figure 14 presents the meannumber of investigations per case at each risklevel. Clearly, the Michigan system did asuperior job identifying families with low,moderate and high proclivities for maltreatingchildren.

Does the Structured Decision Making Model Work? 25

Mean Number of New Investigations and Substantiations Reported During an 18-Month Follow-Up Period by Risk Level

California Michigan Washington

Investigations Substantiations Investigations Substantiations Investigations Substantiations

Low .525 .22 .246 .09 .386 .18

Moderate .658 .28 .541 .21 .665 .27

High .585 .22 .872 .43 .636 .28

Base Rate .580 .24 .586 .25 .596 .25Note: Base rates for each system vary slightly because the number of cases for which each risk

assessment was completed ranges from 929 for Michigan to 876 for California.

Figure 14

Evaluation of the Michigan Structured Decision Making System

Between 1989 and 1992, CRC and Michiganchild welfare staff worked together to design anSDM system for CPS cases (Baird, Wagner,Caskey and Neuenfeldt, 1995). The systemconsisted of risk and needs assessmentinstruments, case planning andreassessment tools, as well asdifferentiated service standards. Systemimplementation began in 13 pilotcounties during 1992.

Did the implementation of this risk-based structured decision making systemhave an impact on child welfareoutcomes? Michigan’s phased implementationschedule presented an opportunity to formally

evaluate the impact ofSDM by comparingoutcomes in the 13SDM counties withthose in a matchedsample of 11 countiesthat were stilloperating under thetraditional system. Theevaluation sampleconsisted of all caseswith substantiated

abuse or neglect between September 1992 andOctober 1993. The SDM and comparison studysamples each totalled approximately 900families. Outcome measures included newreferrals, investigations and substantiationsoccurring during a 12-month follow-up period.

There were several important findingsregarding differences in decision making and

case processing that occurred in the SDM andcomparison counties. The process evaluationfindings included:

Case Closing Decisions

• The SDM counties were significantly morelikely to close low and moderate risk casesfollowing substantiation, while the non-SDM counties closed more high andintensive risk cases. Moreover, cases that wereclosed without services in the SDM countieshad significantly lower re-referral rates thanclosed cases in the comparison group. Thisindicates that the use of risk assessment ledto improved screening in the SDM counties.

Changes in Service Provision

• Program participation in the SDM countiesoccurred at significantly higher levels thanin the comparison counties. This wasparticularly true for high and intensive riskfamilies. For example, high risk families inthe SDM counties were more likely than thehigh risk non-SDM cases to becomeinvolved in parenting skills training,substance abuse treatment, familycounseling and mental health services (seeFigure 15).


High risk families in the SDM counties were significantly more likely to become

involved in parenting skills training,substance abuse treatment, family

counseling and mental health services.

0%

10%

20%

30%

40%

50%

ParentingSkills

SubstanceAbuse Treatment

FamilyCounseling

Mental Health Services

36%

22% 23%18%

40%

25% 24%

14%

SDM Counties Comparison Counties

Michigan SDM Evaluation ResultsPercent of High Risk CPS Cases

that Received Specific Services

Outcomes

The evaluation also examined whether changessuch as those noted above resulted in a betteroverall system of child protection. The principlequestion is whether implementation of theSDM system translated into lower rates ofmaltreatment in Michigan. Figure 16 comparesoverall results for cases from SDM withcomparison counties. For each outcomemeasure, families in the SDM counties hadbetter outcomes than other families. Thegreatest difference was found in rates of newsubstantiations, where SDM families had a ratethat was 50% lower than that observed for thecomparison group (6.2% vs. 13.2%).

A separate analysis of outcomes by risk groupalso showed positive results for the MichiganSDM system. For example, high risk CPS caseshandled in the SDM counties had fewer new

referrals, fewer subsequent child injuries, lowerrates of subsequent placement in foster careand, like the overall sample, were only half aslikely as comparison families to have asubsequent substantiation.

In summary, the results of this carefullycontrolled evaluation showed that SDM notonly resulted in important changes in decisionmaking and service provision for child wel-fare cases but, as anticipated, it ultimately hada positive impact on the protection ofMichigan’s children.

Does the Structured Decision Making Model Work? 27

Figure 15

SDM families had a subsequent substantiation rate that was 50% lower

than that found in the comparison group.


0%

5%

10%

15%

20%

25%

NewReferrals

NewSubstantiations

Removal toFoster Care

Child InjuryReport

18.7%

22.2%

6.2%

13.2%

4.1% 5.2%2.5%

4.1%

SDM Counties Comparison Counties

Michigan SDM Evaluation ResultsOutcomes for CPS Cases

12-Month Follow-Up

Figure 16

Conclusion 29

The future of child welfare servicesin the U.S. depends on the ability ofCPS agencies to effectively deal withgrowing caseloads, increased publicscrutiny, and static or diminishingresources. The number of abuse andneglect complaints has tripled since1980. Clearly, new methods areneeded to deal with this crisis.Agencies cannot ignore technologieswhich significantly improvedecision making and help targetresources to children and familiesmost at risk For example, the use ofempirically-based risk assessment isnot a question of replacingprofessional judgment withstatistical inference. It is simply amatter of using the best informationavailable to protect our childrenfrom harm. And as demonstratedthrough the recent OCAN evaluation, theresearch-based risk tools used by CRC areclearly superior to other models in terms ofboth their reliability and their validity.

Risk assessment is only one component of theCRC system. The CRC model is comprehensive,using structured decision making at all keydecision points from intake to reunification.Moreover, it allows the best information to beused at every organizational level. It linksassessments to service plans, and agencystandards to workload and budgeting. Itprovides data to workers for case decisionmaking and data to managers for planning andprogram evaluation. Finally, as demonstrated bythe OCAN and Michigan evaluation research,the CRC model represents a practical andefficient means for improving the plight ofAmerica’s child protective service systems.

CONCLUSION

30 References

References

Baird, C., D. Wagner, T. Healy and K. Johnson (1999).“Reliability and Validityof Risk Assessment in Child Protective Services: A Comparison of ThreeSystems.” Madison, WI: Children’s Research Center.

Baird, C., D. Wagner, R. Caskey and D. Neuenfeldt (1995). “The MichiganDepartment of Social Services Structured Decision Making System: AnEvaluation of its Impact on Child Protection Services.” Madison, WI:Children’s Research Center.

Dawes, R. (1993). “Finding Guidelines for Tough Decisions,” The Chronicleof Higher Education, June 9, A40.

Dawes, R., D. Faust and P. Meehl (1989). “Clinical Versus ActuarialJudgment,” Science, 243, 1668-1674.

Johnson, W. and J. L’Esperance (1984). “Predicting Recurrence of ChildAbuse,” Social Work Research and Abstracts, 20(2), 21-26.

Meehl, P. (1954). Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of theEvidence. Minneapolis, University of Minnesota Press.

Rossi, P., J. Schuerman and S. Budde (1996). “Understanding Child Maltreatment Decisions andThose Who Make Them.” Chicago: Chapin Hall Center for Children, University of Chicago.

Salovitz, B. (June 1993). “New York State Risk Assessment and Service Planning Model: A Review ofthe Development Process,” Sixth National Roundtable on CPS Risk Assessment: APWA, p. 25.

United States General Accounting Office (1997). “Complex Challenges Require New Strategies,” ChildProtective Services, HEHS-97-115, p. 2.

APPENDIX A

a1 Appendix A

Appendix A a2

a3 Appendix A

0%

10%

20%

30%

40%

50%

60%

Very Low Low Moderate High

8% 2%

18%

7%

36%

20%

54%

34%

*Includes substantiated and unsubstantiated cases.

New Mexico Risk Assessment Study*Outcomes by Risk Level Assigned

New Allegations New Substantiations

0%

10%

20%

30%

40%

50%

60%

70%

Low Low Medium Medium High

6%

23%

63%

39%

Note: Validation study conducted on 956 cases in 1994.

Rhode IslandSubstantiation Rates by Risk Levels

Substantiation Rate

APPENDIX B

b1 Appendix B

“In the last 50 years or so, the question of whether a statistical or clinical approach

is superior has been the subject of extensive empirical investigation; statistical vs.

clinical methods of predicting important human outcomes have been compared with

each other, in what might be described as a ‘contest.’ The results have been uniform.

Even fairly simple statistical models outperform clinical judgment. The superiority

of statistical prediction holds in diverse areas, ranging from diagnosing heart attacks

and predicting who will survive them, to forecasting who will succeed in careers, stay

out of jail on parole, or be dismissed from police forces.

..... objections (to using statistical models) ignore the data from well over 100

studies, almost all of which show the superiority of prediction based on statistics

rather than on experts’ intuition. For example, undergraduate records and test

scores alone predict performance in graduate school better than do the ratings of

admissions committees. The objections to using statistics also ignore the ethical

mandate that, for important social purposes such as protecting children,

decisions should be made in the best way possible. If relevant statistical

information exists, use it. If it doesn’t exist, collect it.” (emphasis added)

- Robin Dawes, Professor

Carnegie Mellon University



www.nccd-crc.orgwww.nccd-crc.org

the improvement of child protective services with structured ......the improvement of child...

Documents