keep them safe outcomes evaluation · keep them safe outcomes evaluation final report – annex j...

Keep them safe OutcOmes evaluatiOn

Final RepoRt – annex J

nSW depaRt m ent oF pRe m ieR and cabine t

June 2 014

liteRatuRe RevieW

Authorsthis report was written by the evaluation team led by the social policy Research centre (spRc) at the university of nsW. the evaluation team includes researchers at the Bankwest curtin economics centre (Bcec) at curtin university, the university of melbourne (uom) and the parenting Research centre (pRc). chief investigators for the Outcomes evaluation are:kylie valentine (spRc)ilan Katz (spRc)Rebecca cassells (curtin)aron shlonsky (uom)

authors of this report include ilan Katz and ciara smyth. suggested citation: Katz i and smyth c (2014), ‘Keep Them Safe Outcomes Evaluation: Literature Review Final Report, Annex J’, sydney: nsW Department of premier and cabinet.

Ethics and police clearancethe Outcomes evaluation has received human research ethics clearance from the university of new south Wales ethics committee. Researchers involved in this project have obtained appropriate clearances (police checks) which are required to work with sensitive datasets.

Keep them safe OutcOmes evaluatiOn

Final RepoRt – annex J

nSW depaRt m ent oF pRe m ieR and cabine t

June 2 014

liteRatuRe RevieW

2

1 introduction 3

2 examples of child protection reforms and evaluation findings from other jurisdictions

5

2.1 system-level reform 5

2.1.1 example1: scotland’s child protection Reform programme and Getting it Right for every child

5

2.2 Differential response 7

2.2.1 example 1: north carolina multiple Response system 8

2.2.2 example 2: Ohio alternative Response pilot project 10

3 challenges associated with evaluating child protection reform efforts/outcomes 13

4 References 15

contents

Keep them Safe OutcOmeS evaluatiOn final Report annex G – literature Review

3

1 introduction

since the 1990s the case for child protection reform has been made in many english-speaking countries whose child protection systems are struggling to address the increasing number of reports being made to child protection authorities and the increasing complexity of many of these cases. in recognition of the crisis confronting many child protection systems there have been calls for ‘a paradigm shift’ in the way that child protection is approached and a range of system failures highlighted. these include:

§§ the ‘highly forensic’, investigative, reactive and adversarial orientation of many child protection systems that focus primarily on risk assessment rather than ‘families in need’;

§§ the highly bureaucratised, risk averse, and process-driven environment in which child protection staff operate; and

§§ fragmented service delivery responses which tend to focus on issues in isolation rather than adopting a collaborative, interagency, systems approach.

in response to these acknowledged system failures it is widely recognised that there is a need to shift from ‘a preoccupation with child protection towards integrated services with the emphasis on prevention, early intervention and family support’ (lonne, parton, thomson, & harries, 2009; munro, 2011; vincent, Daniel, & Jackson, 2010; Waldfogel, 1998).

the nsW Government has made a substantial investment towards protecting children from harm through its $750 million Keep them safe child protection initiative. Keep them safe (hereafter Kts) was introduced in 2009 as the nsW government’s response to the special commission of inquiry into child protection services in nsW undertaken by Justice Wood (Wood inquiry). a detailed description of Kts is provided in section 2.1 of the Outcomes evaluation final Report.

4

this is one of eleven evaluation reports that make up the Kts outcomes evaluation. the evaluation involved nine separate yet complementary methodologies that were designed to address eight evaluation questions and to analyse the various sources of data available to the evaluation team. a detailed description of the evaluation is provided in the Kts Outcomes evaluation final Report.

Keep them safe Outcomes evaluation: final reportannex a. indicators report annex B. unit record analysisannex c. cost effectiveness reportannex D. Workforce surveyannex e. spatial analysis reportannex f. appraisal of project evaluationsannex G. Report on client interviews annex h. Data Developmentannex i. Other nsW strategies and initiativesAnnex J. Literature review

Kts is a large-scale reform, which includes a wide range of initiatives and programs. no comparable system-level child protection reform has been identified through a review of the literature. that being said, several jurisdictions have introduced major reforms to the way that child protection is approached, and in australia most jurisdictions have introduced reforms that are similar to Kts in that they incorporate early intervention and statutory investment and legislative changes . While not on the scale of Kts these reforms have embraced the need for significant changes in child protection and many of the reforms that have been introduced are incorporated within Kts. a number of these reforms have been evaluated, however, given that they are far narrower in scope compared to Kts none is directly comparable with the Kts evaluation. nevertheless, several of these evaluations are based on reforms that are broadly comparable to Kts or subcomponents within Kts and are therefore relevant to this review.


5

2.1 System-level reform

there are few comparable systems reforms of the scale and breadth of Kts, however, we did find examples of other child protection initiatives, the lessons of which are useful for Kts.

2.1.1 Example 1: Scotland’s Child Protection Reform Programme and Getting it Right for Every Child

scotland’s child protection Reform programme (cpRp), a three-year reform program, was initiated in 2003. the goals of the reform program were to improve the protection of children at risk of abuse and neglect and reduce the number of children in need of protection (Daniel et al., 2007). the reform program sought to provide better early support to children and families to prevent them requiring protection at a later stage, provide better direct help to children in need of protection and provide effective continuing support to children and families to prevent child protection concerns recurring (vincent et al., 2010).

among the key initiatives undertaken as part of cpRp were: the development of a children’s charter; the children’s charter was translated into a framework for standards for professionals working in cp and public awareness initiatives. a process review of the cpRp (Daniel, vincent, Ogilvie-Whyte, law, & tarara, 2007) found:

§§ high levels of engagement with stakeholders concerning the reforms

§§ that the cpRp was planned and implemented in a structured way

2 examples of child protection reforms and evaluation findings from other jurisdictions

6

§§ processes introduced have maintained the focus on children’s outcomes

§§ professional advisors were critical to the operation of the cpRp

§§ the cpRp was informed by developmental theory and consistent with policy developments across the uK

§§ the cpRp was successful in articulating the need for a more integrated approach to child care and protection

§§ the stated aims of cpRp were clear

§§ the cpRp has been successful in raising awareness across the board of children’s need for support and protection

§§ the cpRp has contributed to developments in multi-disciplinary initiatives at both strategic and practice levels

§§ the cpRp has resulted in more nursing and education staff providing more direct help to children and making more child protection referrals

§§ the cpRp ‘has developed a significant momentum for change.’

following the completion of the cpRp national policy emphasis in scotland shifted from a narrow focus on ‘child protection’ towards integrated support for children under the Getting it Right for Every Child (GiRfec) reforms. GiRfec does not constitute an overhaul of the child protection system per se, but rather is ‘a set of wide-ranging proposals, relating to children’s services as a whole’ (vincent et al., 2010, p. 441). GiRfec promotes ‘a common, coordinated approach across all agencies that supports the delivery of appropriate, proportionate and timely help to all children as they need it.’ (vincent et al., 2010, p441).

in 2006 an implementation plan was published which outlined a development strategy:

§§ for streamlining children’s records, assessments and action plans,

§§ the development of national practice tools, training materials and guidance,

§§ the development and pilot testing of a prototype electronic solution to facilitate information sharing across children’s services, and

§§ a communication strategy for keeping managers and staff working in children’s services informed of developments.

two pathfinder projects were established with a view ‘to help shape, develop and test the practice tools and training materials and to inform the development of national guidance for Getting it Right.’ (stradling et al., 2009). a review of the development and implementation of GiRfec in one of the two


7

pathfinder project sites drew on data from multiple sources (including interviews, surveys, focus groups and quantitative analysis of administrative data). a limitation of the review was that it was not possible to link the findings to measures of impact on outcomes for children.

the review reported multiple indicators of progress related to improved outcomes for children and families, including a drop in the rate per 1000 of children (0-15) on the child protection Register from 3.0 to 1.5 since 2005; a considerably higher proportion of case conferences leading to registration than in scotland as a whole. the review recommended that the GiRfec approach should be followed in child protection cases ‘which should lead to a more holistic approach to assessment and planning which addressed not only the child’s safety but also his or her other unmet needs.’ the review found that the reforms implemented under GiRfec had led to improvements with respect to changes in professional practice, changes in professional cultures, and strategic and systemic changes.

With regard to professional practice, the review found promising indications. the procedures and pathways followed by professionals were more streamlined and rational; multi-agency collaboration and support were more common, staff were using the same tools, processes and procedures when working with children which was leading to ‘convergence of stronger shared multi-agency thinking and use of language across agencies at each stage of support provision’ (stradling et al., 2009, p. ix). every child in the location under review had a named person in health or education who was responsible for making sure that the child has access to the right help to support his or her development and well-being.

With regard to professional culture, the review found that there was a clear sense of ownership of GiRfec among professionals working with children and young people; the level of inter-agency trust was higher; and there had been a shift in emphasis from measuring service outputs delivered to measuring outcomes for children and young people. signs of progress concerning strategic and systemic change identified by the review included the finding that the vision behind GiRfec was now well-embedded in the review location. in addition, the review found that initial impressions that GiRfec was largely concerned with the most vulnerable children has shifted with professionals grasping that GiRfec was intended to have an impact on universal service provision.

the review also highlighted many issues for further development to ensure GiRfec’s success. these include recommendations concerning changing professional practice, changing professional cultures, developing strategic and systemic change and improving outcomes for children and young people. it also emphasised the importance of a package of support measures rather than one-off training in order to bring about the changes entailed in the GiRfec approach.

2.2 differential response

One of the key changes implemented under Kts was the change in the threshold for reporting children and young people to the child protection helpline from ‘Risk of harm’ (ROh) to ‘Risk of significant harm’ (ROsh). in response to this threshold change a dual track response has been implemented whereby cases in which children and young people are deemed at ‘risk of significant harm’ generate a statutory response from the nsW Department of community services. cases falling below the ‘risk of significant harm’ threshold are not deemed to require a statutory response but are recognised

8

as being in need of assistance and support either from nsW Department of community services or another agency.

this dual track response implemented through Kts corresponds with the ‘differential response’ track implemented in several jurisdictions alongside the traditional investigative child protection response. Differential response (also called ‘dual track’, ‘multiple track’, ‘family assessment response’ or ‘alternative response’) is an acknowledgement that not all child protection reports require the same response and that a ‘one size fits all’ approach is ineffective in addressing the needs of all who come in contact with the child protection system. the two major paths of response are investigation and assessment, with cases deemed to be low or medium risk proceeding along the assessment track. through the assessment track, efforts are made to identify families’ strengths and needs and address issues that compromise child safety ‘with the goal of avoiding more serious maltreatment and/or future interaction with child protection systems’ (lawrence, Rosanbalm, & Dodge, 2011).

citing a report by Kempe center for prevention of child abuse and neglect (2012), fluke, merkel-holguin, and schene (2013) report that 17 states in the us have state-wide differential response programs, five states have multi-region programs and 12 are ‘developing a Differential Response-organized child protection system’. Differential response is defined and implemented differently in different jurisdictions (hughes, Rycus, saunders-adams, hughes, & hughes, 2013; lawrence et al., 2011) several of which have conducted evaluations to assess the effectiveness of this policy reform.

2.2.1 Example 1: North Carolina Multiple Response System

north carolina implemented a differential response track alongside a seven-part plan as part of their larger multiple Response system (mRs) child protection reform. the goal of mRs is ‘to respond not only to the specific incident that brought a particular family to the attention of the child protection agency, but to understand and address the broader spectrum of needs that might have undermined the caregivers’ ability to parent effectively.’ (lawrence et al., 2011). in 2002 10 counties were selected for pilot implementation. Based on favourable evaluation findings mRs was implemented state-wide in 2006. no additional funding was allocated to counties to implement mRs reforms.

the mRs reforms entail a family-centred approach that incorporates seven key strategies:

§§ choice between two approaches to reports of child abuse, neglect or dependency

§§ utilisation of child and family team meetings to make decisions

§§ collaboration between Work first (welfare-to-work) and the child welfare program meetings as appropriate

§§ implementation of a strengths-based, structured intake process

§§ Re-design of in-home services


9

§§ implementation of shared parenting meetings (between birth parents and foster parents) in cases where a child has been placed in out-of-home care

§§ coordination between law enforcement agencies and child protective services for the investigative assessment approach.

the outcomes evaluation findings point to a favourable impact of mRs on child safety when pilot county data were compared with control county data. the implementation of mRs was associated with:

§§ lowered mean rates of substantiated maltreatment cases

§§ a decline in substantiation rates over time

§§ a decline in maltreatment re-assessment (investigation) among those families that had been assessed previously.

timeliness in commencing assessment dropped initially, which the authors suggest is related to workers learning to adapt to the new system, later returning to previous levels as staff adapted to the changes.

the process evaluation found an increase over time in the proportion of cases handled using family assessment – from fewer than 40 per cent in the first year of mRs implementation (2002), to 80 per cent in 2008. feedback from staff suggested that the implementation of child and family teams had improved social work practice with multiple benefits identified in addition to several ongoing challenges. staff also held very positive views concerning collaboration with other community-based service providers.

lawrence et al. (2011) report that two-thirds of the comments concerning families’ experience with cps were positive. additional findings from phone interviews with 223 caregivers were:

§§ 31% reported that the services improved their parenting skills;

§§ 28% said that it helped them to better deal with conflict;

§§ 21% said it helped them to know who to contact in the community should they need assistance in the future;

§§ 32% said it helped them to better provide for their family’s needs; and

§§ 1% said it helped them to feel better about themselves and their family.

in their summation of the effectiveness of mRs, lawrence et al. (2011) cautiously conclude ‘that children are no less safe than before mRs, and may even be safer. the findings for recidivism support this conclusion.’ (2011).

10

2.2.2 Example 2: Ohio Alternative Response Pilot Project

the Ohio alternative Response pilot project commenced in 10 counties in 2008. corresponding with differential response, alternative response provides a non-adversarial family assessment process for addressing child protection reports. an initial evaluation report was completed in april 2010, however, given the short time-frame for follow-up data collection the evaluation was extended and an additional 3.5 years of data were collected and a final report completed in september 2013 (loman & siegel, 2013).

for the purpose of the evaluation, families who were deemed appropriate for an alternative response family assessment were randomly assigned to experimental (family assessment) and control (investigated) conditions. Key evaluation findings include:

§§ significant improvements in family engagement occurred under alternative Response (aR) with the authors asserting that ‘the increase in positive feelings and in family engagement are not the consequence of having received funded services but are the result of the approach to families taken by workers under aR’

§§ among staff who had experience with aR cases or supervision over two-thirds felt that children were equally safe under aR and traditional investigations and another tenth felt children were safer

§§ the majority (95%) of staff members involved in aR felt that their knowledge of the goals and philosophy of aR was adequate or thorough

§§ Between two-thirds and three-quarters of staff felt that families on the aR track were more likely to cooperate and participate in decisions and case plans compared to families in the traditional child protection track

§§ the evaluation found that in the follow-up period of four to five years the experimental families who initially received an aR family assessment exhibited statistically significant lower levels of five types of subsequent child safety problems when compared to control families. these were: serious inflicted harm, danger from a mentally or physical ill adult, lack of supervision or failure to provide basic needs, refusal of access to the child or likelihood of family flight and failure to meet serious physical or mental health needs of the child

§§ in the follow-up period experimental families who initially received an aR family assessment were less likely to experience problems associated with parenting

§§ experimental families who initially received an aR family assessment had significantly fewer new accepted (screened-in) reports of child maltreatment. this effect was found primarily among lower-risk families who had their first contact with child protection system around the same time they entered the study

§§ experimental families who initially received an aR family assessment experienced lower levels of child removal and out-of-home placements compared to families in the traditional child protection track


11

§§ the evaluation found modest cost savings for families in the aR track compared to families on the traditional child protection track.

While these evaluations of differential response models suggest that these reforms are showing some early promising signs the models are not universally endorsed. hughes et al. (2013)’s meta-evaluation of differential response programs in a number of us jurisdictions concluded that there was inconsistency in differential response implementation with no uniform, standardized practice model to follow; that methodological problems in differential response research and evaluation limit confidence in research findings and conclusions; and that there is insufficient evidence to confirm the safety of children served in alternative tracks. in their response to hughes et al. (2013)’s paper fluke et al. (2013) accepted a number of their criticisms whilst challenging other assertions. for fluke et al. the shift from an adversarial, fault-finding orientation to a strengths-based, needs-based approach evident in differential responses is a positive development and that ‘the notion that differential responses or any other systems level reform should not be implemented until the highest standard of research is conducted is unrealistic’.


13

achieving culture change in human services takes time (Glisson, Dukes, & Green, 2006). in his commentary concerning Glisson et al’s paper, nunno (2006) focuses on the findings concerning organisational culture, making the claim that effecting organizational culture change can take five to seven years. Yet, the review of the implementation of the GiRfec reforms in scotland was suggestive of changes in professional cultures in a shorter time frame, stating that ‘changes of this magnitude and scope take at least three years before they are embedded in the practices of the majority of staff’. however, the report emphasises that achieving a shift in professional cultures requires effective induction programs for new staff (stradling et al., 2009).

3 challenges associated with evaluating child protection reform efforts/outcomes


15

Daniel, B., vincent, s., Ogilvie-Whyte, s., law, J., & tarara, h. (2007). A process review of the child protection reform programme. edinburgh: scottish executive.

fluke, J. D., merkel-holguin, l., & schene, p. (2013). thinking differentially: a response to issues in differential response. Research on Social Work Practice, 23(5), 545-549.

Glisson, c., Dukes, D., & Green, p. (2006). the effects of the aRc organizational intervention on caseworker turnover, climate, and culture in children’s service systems. Child Abuse & Neglect, 30(8), 855-880.

hughes, R. c., Rycus, J. s., saunders-adams, s. m., hughes, l. K., & hughes, K. n. (2013). issues in differential response. Research on Social Work Practice, 23(5), 493-520. doi: 10.1177/1049731512466312

lawrence, c. n., Rosanbalm, K. D., & Dodge, K. a. (2011). multiple response system: evaluation of policy change in north carolina’s child Welfare system. Children and Youth Services Review, 33(11), 2355-2365.

loman, l. a., & siegel, G., l. (2013). Ohio alternative response evaluation extension: Final report to the Ohio Supreme Court. st. louis, missouri: institute of applied Research.

lonne, B., parton, n., thomson, J., & harries, m. (2009). Reforming Child Protection. london: Routledge.

munro, e. (2011). The Munro review of child protection: Final report, a child-centred system. united Kingdom, Department for education.

4 References

16

nunno, m. (2006). invited commentary. the effects of the aRc organizational intervention on caseworker turnover, climate, and culture in children’s services systems. Child Abuse & Neglect, 30(8), 849-854.

stradling, B., mcneill, m., & Berry, h. (2009). Changing professional practice and ulture to get it right for every child an evaluation of the development and early implementation phases of Getting it Right for every child in Highland: 2006-2009. edinburgh, scottish Government.

vincent, s., Daniel, B., & Jackson, s. (2010). Where now for ‘child protection’ in scotland? Child Abuse Review, 19(6), 438-456.

Waldfogel, J. (1998). Rethinking the paradigm for child protection. The Future of Children, 8(1), 104-119.

keep them safe outcomes evaluation · keep them safe outcomes evaluation final report – annex j...

Documents