hevesi, foster care outcomes and performance standards, 1997

7/31/2019 Hevesi, Foster Care Outcomes and Performance Standards, 1997

1/76

Foster Care Outcomes andPerformance Standards

A Report of the Task Force onFoster Care Performance Standards

City of New York

Office of the Comptroller

Office of Policy Management

Alan G. Hevesi

Comptroller

Barbara Salmanson

Senior Policy Adviser

Task Force Chairpersons

Barbara BlumNational Center for Children in Poverty

David Liederman

Child Welfare League of America

Fred Wulczyn, Author

Chapin Hall Center for Children

May 19, 1997


2/76

Barbara Blum is currently Senior Fellow in Child and Family Policy at the National Center forChildren in Poverty at Columbia University. From 1986 through 1996, she was president of theFoundation for Child Development where she focused her energies on improving programs and

policies affecting children and families at risk. Her other past positions include president of theManpower Demonstration Research Corporation and Commissioner of the New York State

Department of Social Services. She also serves as chair of the Agenda for Children Tomorrow,and is a member of the boards of directors of the March of Dimes and the Academy forEducational Development and a member of the New York State Commission on Child Abuse andNeglect.

David Liederman, since 1984, has been the Executive Director of the Child Welfare League of

America, the largest voluntary organization in the U.S. concerned with improving services toabused and neglected children and their families. He has been a strong advocate on behalf ofchildren during his two terms in the State of Massachusetts House of Representatives, as

Commissioner of the Massachusetts State Office of Children, as Chief of Staff to Governor

Michael Dukakis and as Executive Director for Public Affairs for the Federation of JewishPhilanthropies of New York.

Fred Wulczyn is currently a research fellow at the Chapin Hall Center for Children at theUniversity of Chicago, where he oversees Chapin Halls Managed Care Forum and co-directs themultistate foster care data archive. Prior to joining Chapin Hall in 1997, he was the Director of

the New York State Department of Social Services Managed Care Initiative. He is also amember of advisory committees for the National Study of Kinship Foster Care and the NationalEvaluation of Family Support and Family Preservation Programs.

Alan G. Hevesi

Comptroller

Steven Newman


3/76

First Deputy Comptroller

Jeanne Millman

Director

Barbara Salmanson

Project Manager

Office of Policy Management

!

Fred Wulczyn

Report Author

Steven Budde

Research Associate

Chapin Hall Center for Children

Comptroller's Task Force on Foster Care Performance Standards

Co-chairs

BarbaraBlum,National Center for Children in Poverty

David Liederman, Child Welfare League ofAmerica

Members

Lynette Brown, Council of Family and Child Caring Agencies

Steven Cohen, Jewish Board of Family and Children's Services

John Courtney, Child Welfare Watch

Mario Drummonds,Northern Manhattan Perinatal Partnership

Judith Hines, Council on Accreditation of Services for Families and Children

Donna Lawrence, Children's Defense Fund

Elba Montalvo, Committee for Hispanic Children and Families

Sr. Mary Paul, Center for Family Life

Elizabeth Schnur,Jewish Child Care Association

Janice Shindler,Association of Black Social Workers

Michael Solomon,Ncw York State Comptroller's Office

Fred Wulczyn, Chapin Hall Center for Children


4/76

May 19, 1997

any thanks to the members of the Task Force on Foster Care Perfor-mance Standards for their commitment and hard work which culmi-

nated in the publication of this report. Their efforts to refocus

Mattention on how the foster care system affects the lives of childrenand families will improve the condition of foster children and force greater account-ability among social service systems serving them and their families. I recognize thatthis cannot be accomplished overnight. However, this report presents a major firststep in that very important process.

I express my profound gratitude to both Barbara Blum and DavidLiederman, co-chairs of the Task Force who led this very important effort. Theirsupport and direction have been invaluable.

I would like to give a special thank you to Fred Wulczyn, author of thereport, for his hard work, expertise and guidance on these complex issues, andacknowledge the Foundation for Child Development for its financial support ofFreds efforts.

I also express my apreciation to Commissioner Nicholas Scoppetta and theAdministration for Childrens Services for their commitment to improving thequality of foster care services and their encouragement and ongoing support for thework of the Task Force. We look forward to continuing to work closely together toImprove the quality of services for New York Citys children.

Finally, I thank Barbara Salmanson for managing this important project

through to its completion, and David Neustadt and Mary Leopold for their editorialassistance.

Alan G. HevesiComptroller


5/76

Highlights

The purpose of a child welfare system is to identify children in dangerous situations and provide

the appropriate protections. Oftentimes, this means moving them to temporary foster homes and,as quickly as possible, provide permanent, safe homes. The foster care system should be assupportive as possible to children who are going through an inherently difficult and distressingexperience. The safety and well-being of the children must be the paramount goal.

As obvious as that sounds, foster care agencies are not currently evaluated on how well they takecare of children. Current standards for evaluating agencies focus on process and inputs rather thanoutcomes. Data is gathered on whether agencies meet timetables for moving children to adoptionand filing reports with the Administration for Childrens Service's, but not on the experience ofchildren during the time they are in foster care.

As a result, each year hundreds of millions of public dollars are spent in New York City withoutobtaining a true picture of the impact the foster care system has on the children living within it.Consequently, no accurate evaluation can be made nor can change be implemented to improve the

experiences of these vulnerable children.

The Comptroller's Task Force on Foster Care Performance Standards is working to refocus theevaluation process by helping to create new performance standards focussed on outcomes and on

children and their families. In this report, the Task Force has developed principles that shouldguide the development of these new child-centered standards.

The report mandates accountability based on how the system is succeeding in meeting children's

needs in three major areas:

1.System Performance: Safety and Permanence,2.Child and Family Well-being, and

3.Client Satisfaction.

The report and its recomendations identify the outcomes that should be the focus of the work offoster care agencies and the individual indicators which measure progress towards achieving thoseoutcomes.

The report focuses attention on child and family well-being indicators which address the physical

and emotional state of the child and look at children in the context of their families. For the firsttime, client satisfaction is being treated as a valuable measure in determining the quality ofservices provided. Although a family member's level of satisfaction with a service or provider maynot be a definitive measure of service quality, it does serve as a valuable resource in understandingand evaluating the effects of services on children and biological and foster parents.

i


6/76

Foster children are involved with a variety of social service systems. Too often, the buck ispassed from one system to the other, and children's needs go unmet. The report recognizes thecomplexities and interplay between and among the various social service systems and attempts tointegrate them to obtain a comprehensive picture of a foster child's experience. This new

approach will help the Administration for Children's Services impose accountability on the child

welfare system to meet the needs of foster children.

As a result of the implementation of such a system, the nature of evaluations of the performance

of foster care agency performance will be more qualitative, comprehensive and child-focused.It will give the system greater knowledge about the strengths and weaknesses of program designand the successes and failures of specific programs. Most importantly, it will base performanceon what is done for the child. As a result, we hope to achieve greater accountability so the quality

and effectiveness of services to children continues to improve, along with casework practice,clinical decision making and planning and resource allocation both externally and internally. In thisway, limited flinds can be used in the manner best suited to meet the needs of children and familiesto assure child safety and family integrity.

ii


7/76

Contents

page

Highlights

Part One: Introduction and Overview i

I. Introduction 3II. Perspectives on Foster Care Outcomes and Performance Standards 5

III. Overview of Recommendations 8System Performance: Child Safety and Permanency 10Child and Family Well-Being 11Client Satisfaction 11

Part Two: Task Force Recommendations 13

IV. Domain One: System Performance: Child Safety and Permanency 14Safety and Permanency Issues 14Length of Stay 14Recommended Outcome Measures 15Proposed Tools for Outcome Measurement 16Phase One: Use Existing Resources 17Phase Two: Link Information 20

V. Domain Two: Child and Family Well-Being 23The Challenge of Measuring Well-Being 23Recommended Outcome Measures 25Proposed Tools for Outcome Measurement 29Phase One: Undertake a Study of Child and Family Well-Being 29Phase Two: Create a Working Group 30

VI. Domain Three: Client Satisfaction 32The Importance of Client Satisfaction Measures 32Recommended Outcome Measures 33Proposed Tools for Outcome Measurement 35Phase One: Implement Guidelines and Surveys 35Phase Two: Establish an Advisory Panel 37

AppendicesA. Methodological and Conceptual Issues that Arise in Conducting Research on

Child and Family Well-being

B. Methodological and Conceptual Issues that Arise in Measuring Client Satisfaction

C. Summary of 1996 Focus Group with Parents Regarding Client Satisfaction

D. References


8/76

2

Part OneIntroduction and Overview


9/76

3

I. Introduction

Recent tragedies have highlighted the need to improve the performance of New York Citys child

welfare system. A vital part of that process must be the development and implementation of a

system of outcomes and performance measures that describes what happens to children and their

families when a child enters the foster care system. In short, we need to know what works, what

does not, and why. Current measures tend to focus on process rather than results, on inputs rather

than outcomes, on how much is spent rather than what is accomplished, on quantity rather than

quality. Absent a more coherent approach to fundamental questions regarding how well children

are served by the child welfare system, the system improvements long hoped for will remain a

remote possibility.

The renewed emphasis on child welfare outcomes comes at an important time for other

reasons. The mission of the child welfare system - ensuring the safety of children whilepreserving the integrity of families - is become increasingly difficult to operationalize, let alone

fulfill. The federal policies that delineate public responsibility for needy persons, including

children, are in the midst of a transformation that will fundamentally alter Americas social safety

net. For their part, some states around the country have already enacted time limits for welfare

recipients, block grants for child welfare services, and managed care for health and behavioral

health care, initiatives that are often accompanied by significant reductions in funding and a

devolution of authority in the direction of local government. Enacted with an eye toward making

government more responsive to local conditions, the changes stand to have a profound impact on

child welfare systems. In short, at a time when the need for child welfare services is as high as it

has ever been, the very service delivery system designed to meet that need is under enormous

pressure to function more effectively and more efficiently.

In a policy and fiscal context that demands the effective and efficient delivery of services,

no organization, public or private, can be expected to improve on its performance without clear

outcomes and appropriate performance standards. To this end, the analysis of program data, much

of which is already routinely gathered, fulfills an essential purpose by providing key insights into

what is happening to children and families. Long a staple of effective management within industry,

government and not-for-profit agencies are just now learning how to use program data

strategically. Within this context, outcome measures specify the results toward which resources

must be allocated, while performance standards provide a baseline for comparing accomplishments

with expectations. Together, outcome measures and performance standards provide an essential

framework for feedback and accountability, making it possible to evaluate critically the decisions

made by government agencies and the providers who serve the public.


10/76

4

Under Alan G. Hevesi, the New York City Comptrollers Office has made improved

performance monitoring of government programs one of its priorities. As part of that effort, the

Comptroller convened a Task Force on Foster Care Performance Standards in July 1995 to

develop measures based on outcomes achieved for children in care and their families. In

completing its work, the Task Force set out to develop both a reasonable set of indicators and

appropriate methods that could be used to monitor New York Citys child welfare system. Over

a series of meetings held since the late summer of 1995, the Task Force reached agreement on

the need for outcome measures and performance standards; the criteria to be used in selecting

indicators; and the specific measures and standards that underlie its recommendations. Briefly

the recommendations fall into three domains:

1. System performance: child safety and permanency,

2. Child and family well-being, and

3. Client satisfaction.

The complexity of the problems families face and the tension between family preservation and child

protection can undermine the singular sense of purpose that leads to clear outcome statements and

performance standards. As such, the outcome domains are not discrete. On the contrary, an

effective system of outcome measurement and monitoring depends on the ability to draw a

comprehensive portrait of how well the system is performing and how well children and families

are thriving using together all three outcome domains that were examined by the Task Force.

The Report addresses these issues in depth and proposes ways to develop a coherent plan

with an eye toward improving upon the state of the art in child welfare. It begins with an overall

orientation to outcomes and performance standards, enumerates their uses, and describes the

criteria that were used in selecting them. The specific recommendations of the Task Force

follow. In addition to specifying outcomes in each of the three domains, it describes the

relationship between the outcomes and the mission of the child welfare system, the information

burden (cost) associated with collecting information systematically, and other important

methodological or resource considerations that govern the use of given sets of indicators.

Finally, the Report details long-term recommendations that go beyond the specific outcome

measures and performance standards. This second set of recommendations is intended to provide

an overarching framework for improving the capacity of New York Citys child welfare system

to not only gather outcome data, but also use it effectively.


11/76

5

II. Perspectives on Foster Care Outcomesand Performance Standards

Although use of the term outcome is increasingly commonplace within child welfare, the term is

seldom defined with real precision. The diversity of children and their families, the problems they

present, and the settings in which they are served make it difficult to identify a common set of

outcomes that are applicable to each and every child served by the foster care system.

Consequently, the best place to start is with the core missions or purposes of the child welfare

system: safety and permanency.

Child safety is the first objective of any child welfare system. The threat to safety is the

reason for the systems involvement in the familys life and the primary problem to be

solved.

Permanency of placement, either through reunification or adoption, is the mandated goal

for children in out-of home care and the simplest measure of a systems success.

Promoting the welfare of children through stable placements is part of this core objective.

These two broad purposes provide the beginning framework for formulating performance

standards. Nevertheless, because of the range and complexity of possible outcome indicators, the

Task Force had to refine its approach by identifying the purposes of the performance standards,

recognizing that the standards themselves will necessarily evolve with experience. Thus, in the

view of the Task Force, outcome data and performance standards should at a minimum:

1. Provide more accountability than prevailing standards, so the quality andeffectiveness of services to children and families continues to improve;

2. Guide casework practice and clinical decision making;3. Improve policy analysis;4. Enhance public agency management, planning, and resource allocation;5. Enhance contract agency performance, including planning and resource allocation;

and6. Lay the groundwork for a cumulative record of outcomes and performance to

which detail can be systematically added over time.

The selection of appropriate outcome measures should be guided by these purposes. The

process should also be guided by a pragmatism that recognizes the strengths and weaknesses of

the existing system of outcome measurement and takes into account how outcome data are actually

used. Since there are so many potential outcome measures, understanding their end uses helps to

establish priorities. The Task Force chose these criteria for selecting indicators:


12/76

6

1. Performance standards should be consistent with existing statutes and regulationsas they pertain to utilization review standards, the time it takes to reunify childrenwith parents, and the time it takes adoptions to be finalized.

2. Outcome measures must relate to the two core missions of the child welfare system:

safety and permanency.

3. The list of outcomes should reflect the developmental status of the child and thefamily and their history within the system.4. Outcomes must be measurable with current information technology. The measures

should have a historical baseline. Where no baseline data exist, a long-termcommitment to establishing baseline data must be made.

5. Outcomes should be evaluated in light of the ease of tracking the data needed tounderstand the performance of the child welfare system as a whole and its impact onchildren and their families. Data requirements that add significantly to theinformation burden should be integrated slowly unless there is a compelling reasonto move fast.

Finally, though it may loom large, foster care is but one influence in the lives of childrenand families. Properly weighing its effect on human development is a complicated undertaking

under even the most controlled conditions. Our expectations for what a system of outcome

measurement can accomplish should be driven by a common desire to provide the best possible

services to children and families, tempered by an appreciation for the limits of information

technology. Consequently, careful attention must be paid to questions that pertain to how outcome

data can be best analyzed to yield reliable conclusions about system performance and to whether

these conclusions can be generalized to sub-populations within the system. One crucial precaution

has to do with the conventional snapshot approach to measuring outcomes. Even if it is used

repeatedly over time, the snap shot approach to outcome measurement has only limited

applicability because it tends to obscure real changes in the experiences of children and families.

Our methods must also be able to distinguish between the appropriateness of the indicator itself

and the difficulty of gathering, interpreting, and/or analyzing the data. Careful attention must also

be paid to how each measure is going to beusedin conjunction with the others to enhance our

overall understanding of clinical efficacy and system performance.

With these purposes and criteria in mind, we returned to the three broad domains of outcome

measures and performance standards:

1. System performance: Child safety and permanency,

2. Child and family well-being, and

3. Client satisfaction.

The discussion of each domain is structured as follows. The section on each domain


13/76

7

begins with an overview and a description of the areas covered, continues with an explanation of

how the specific indicators within this domain relate to the mission of the child welfare system, and

concludes with proposed indicators and measurement tools. In addition to the main body of the

report, the Appendices provide more detailed assessments of the methodological difficulties

associated with studying child and family well-being and client satisfaction. The Appendices also

contain the results of a focus group discussion with parents of children in foster care that was used

as an orientation to the broader issues of client satisfaction. Finally, the bibliography at the end of

the report offers a list of references that should prove useful to anyone interested in pursuing these

topics in greater detail.


14/76

8

III. Overview of Recommendations

The outcome measures selected by the Task Force were culled from a crowded field of potential

performance indicators. The indicators that were selected were chosen because they are linked

directly to the core mission of child welfare and they have relatively light data collection burdens.

Thus, they do not add significant new costs, at least in the short term.

For the first domain, system performance: child safety and permanency, the indicators

selected are direct measures of safety and permanency. Monitored over time, they provide a basic

understanding of how the system is serving the children in its care. Some of the indicators

measure child safety as demonstrated by incidence or type of maltreatment. Others measure the

systems ability to provide continuity and stability for the children in its care by collecting data on

length of stay, number of placements, or timing and frequency of re-entry into the system.

The second domain, child and family well-being, presented the greatest challenge to the

Task Force. There is no single set of well-being indicators that fits all the children the foster care

system serves. Moreover, it is hard to isolate the impact of foster care placement from other

factors in a childs or a familys experience. Finally, despite the broad appeal of well-being

indicators, the link between well-being and the mission of the foster care system must be carefully

established. Well-being indicators should not be left out of a systematic performance review, but

neither should they be stretched to include outcomes beyond the mission of the child welfare

system.With these considerations in mind, the Task Force developed child and family well-being

measures organized around the central mission of foster care: child safety. Thus, the incidence of

child maltreatment and measures of the childs health, emotional and behavioral status, educational

development, and competencies and achievements can be used to demonstrate child well-being.

Other indicators relating to child safety should be collected. These include measures of family

safety, such as incidence of domestic violence, and of the familys ability to meet the basic needs

of the child. Parental education, level of interaction with the child, drug and alcohol use, emotional

health, and the quality of the familys social supports should also be measured.

To develop recommendations for the third domain, client satisfaction, the Task Force

convened a focus group of parents who had children in foster care either currently or in the past.

From these conversations, four areas emerged as particularly relevant to New York Citys foster

care system. They are:


15/76

9

1. overall satisfaction,

2. satisfaction with the workers providing services,

3. satisfaction with specific services, and

4. satisfaction with communication.

Within each area, the Task Force suggests specific indicators to measure client satisfaction.

Data Collection and Costs

Even the best designed indicators will not provide the desired information if data is too hard to

collect. Conventional wisdom holds that the facts needed to monitor the performance of the child

welfare system are unavailable. But many jurisdictions, including New York City, collect vast

amounts of data about children in foster care including the dates of placement, the agency withcustody, the childs legal status, permanency placement goals, and other related information. This

report presents a series of steps for gathering the required information, categorized into immediate

and long-term phases. The Task Force suggests working with whatever reliable data already exist

in the initial phase, then collecting information later on measures for which reliable data is now

scarce.


16/76

10

The Recommendations in BriefSystem Performance: Child Safety and Permanency

Phase 1

Report to the Public: The Administration for Childrens Services should use currently collected

data to produce an annual report to the public on the dynamics of New York Citys foster care

population. The report would:

1. present the outcome measures the Task Force has recommended for evaluating thisdomain,

2. cover child safety and permanency outcomes for specific sub-populations ofchildren, such as those who remain in care for long periods of time,

3. summarize statistics on indicators of data quality, such as reporting lag andnumbers of corrections required, and

4. describe changes, if any, made to the conclusions from previous reports inresponse to record corrections.

Monitor Contract Agency Performance: The Task Force recommends using internal indicator

reports that track child safety and permanency outcomes by contract agency, as well as the

Administrations own direct care caseload.

Phase Two

The Administration for Childrens Services should add new information to its system and link with

existing data resources. This may be accomplished in three ways.

1. by redesigning data systems to include more information about clients, theirfamilies, and the services they receive,

2. by using case readings or other survey methodologies to expand the scope ofavailable information, and

3. by linking records with existing sources of computerized administrative data, suchas vital statistics records, to form a more comprehensive picture. Notwithstandingthe critical issue of confidentiality, record linkage can be an important tool forunderstanding foster care outcomes.


17/76

11

Child and Family Well-Being

Phase One

Like client satisfaction, this domain has a high information burden. The child welfare

system does not currently track well-being indicators. Existing instruments may not be adaptableto this purpose, and baseline data on well-being outcomes is limited. Thus, in Phase One the Task

Force proposes that ACS:

1. undertake a highly focused study of child and family well-being, based on arandom sample of foster children, that stresses child safety in a permanencyplanning context, and

2. if possible, model the primary data collection instrument on the safety assessmentcurrently in use.

Phase Two

In Phase Two, ACS should establish a working group to develop a long-range plan and a time line

for integrating child and family well-being measures into the performance evaluation plan. The

working group should consider:

1. instrument development,

2. the role of contract agencies in conducting routine studies of well-being, and

3. the creation of reporting standards to permit comparative analysis of agencyperformance.

Client Satisfaction

Phase One

Child welfare agencies currently gather little or no systematic data on client satisfaction. Phase One

would make measures of client satisfaction a regular form of feedback within the child welfare

system. The Task Force recommends that ACS:1. develop a set of uniform questions on client satisfaction, and

2. conduct surveys of client satisfaction modeled on the rapid assessment technologiesused in the health and behavioral health care sectors and involving small, well-targeted samples of children.

Phase Two

The Task Force also recommends long-term collection of information on client satisfaction. ACS


18/76

12

could establish an advisory panel to address the cost and technical issues related to measuring client

satisfaction across representative populations and related service providers. The panels tasks

would include:

3. selecting specific client satisfaction outcome measures,

4. establishing standards for assessment, and

5. developing a plan to include a broad array of clients.


19/76

13

Part TwoTask Force Recommendations


20/76

14

IV. Domain One:

System Performance: Child Safety and Permanency

When children are removed from their family home, government officials have a fundamental

obligation to provide services that conform with public policy and sound social work practice.

With respect to public policy, the child welfare system is governed by generally accepted

principles, which in turn lay the groundwork for judging system performance. These principles

are firmly rooted in the concepts of child safety and permanency planning.

Safety and Permanency

Safety refers simply to the fact that children should not be subject to abuse or neglect, as

defined in public law, while in the care and custody of those who have legal responsibility

for them. This includes the times when a child is in the care and custody of a local social

services commissioner and in out-of-home care.

Permanency is a somewhat more difficult term to operationalize. Public policy currently

includes a strong preference for placement prevention because parent/child separation can

be so traumatic. In addition, because foster care is intended to be a temporary solution,

placements should be brief unless discharge from foster care would endanger a child. If

family reunification does not serve the childs best interests, adoption is considered to bethe next best permanency plan.

Length of Stay

Few statistics are more important than length of stay in creating an overall impression of system

performance. However, interpreting this seemingly simple measure is often quite complex.

Permanency planning is the overarching framework for foster care outcome measures, but child

safety is the reason for intervening in the first place. Measuring length of stay includes tracking

childrens movements into, out of, and back into foster care, between foster homes, and between

levels of care.

Because children thrive best when they maintain stable relationships with their caregivers,

repeated, unplanned movements between a parents home and the foster care system have to be


21/76


22/76

16

Table 1.

Domain One

System Performance Safety and Permanency

Areas of System

Performance Indicators Issues to consider

Child safety 1. Incidence of maltreatment 1. Applicable to children at home, in

care, and after discharge.

2. Type of maltreatment. 2. Indicators should be used in

conjunction with other indicators.

3. Injuries due to maltreatment. 3. Patterns should be tracked over

time.

4. Harm by peers in placement or 4. Measures are relevant at the

siblings at home. individual and aggregate levels.

Continuity and stability of 1. Length of stay. 1. Relationship between length of

care: permanency stay, reentry, and subsequent

maltreatment is crucial.

2. Number of placements including 2. Patterns must be studied within

level of care. sub-populations, over time, and withcommunities.

3. Reentry rates and timing. 3. Measures are relevant at the

individual and aggregate levels.

4. Exit rates by type of discharge.

Proposed Tools for Outcome Measurement

In many jurisdictions, it may be true that the information needed to monitor the performance of

the child welfare system is not available. However, some jurisdictions, including New York City,collect vast amounts of data about children in foster care, their legal status, permanency planning

goals, movement histories, and related information. Even so, misperceptions about the availability

of information persist and are confounded by concerns about data quality. This is a fundamental

issue, especially when the information is used to make normative judgments about system

performance. The Task Forces recommendations for this domain address these concerns.

Since New York Citys data holdings are relatively rich, the Task Force opted for Phase


23/76

17

One recommendations that would maximize the usefulness of existing data rather than calling for

costly, new data collection strategies. Two facts influenced this decision.

1. The New York State Department of Social Services is implementing a large-scalecomputer project, the Connections Project, that will substantially alter how

information is collected and used in child welfare service delivery.2. There is room for improvement in the way information is presented and

subsequently used. The integration of information into decision-making has beenhampered by a poor understanding of how foster care tracking data can be used tointerpret even simple measures like length of stay.

Since the Task Forces recommended outcome indicators already appear frequently in

management reports, the recommendations of the Task Force for Phase One amount to guidelines

for reporting data.

Phase One: Use Existing Resources

The Administration for Childrens Services already collects the data it needs to monitor the Task

Forces recommended indicators for safety and permanency. Therefore, the emphasis in Phase

One is on maximizing the use of those data without major new investments. Recommendations

focus on reporting existing data to create the most accurate possible portrait of what happens to

children placed in foster care. While the information burden is relatively low, costs arise from

manipulating existing databases, like the Child Care Review Service (CCRS) and the State Central

Register (SCR).

During Phase One, the Task Force recommends that ACS use existing data to produce an

annual report giving information on the specific outcome measurements, using admission cohorts

as the foundation for the analysis. The report should include a separate analysis of child safety and

permanency related outcomes for specific sub-populations of children. In addition, the report

should include summary statistics on indicators of data quality. The Task Force also suggests that

in Phase One, data be used to monitor contract agency performance for internal ACS review.


24/76

18

Annual Report: The Dynamics of New York Citys Foster Care Population

While recognizing that the Administration for Childrens Services already publishes numerous

reports describing the experiences of children in foster, the Task Force nevertheless recommends

that the Administration produce an annual report describing the dynamics of New York Citys

foster care population. The data presented in the report should be based on entry cohorts of

children, treating children admitted for the first time separately from children re-entering placement.

Research studies based on administrative data have repeatedly demonstrated the shortcomings of

point-in-time or snapshot samples of children in foster care. Thus data from these samples

often lead to erroneous conclusions about the experiences of children who are placed in foster care.

A report that describes the experience of successive cohorts of children should correct for these

problems and substantially improve what is already known about the placement experience ofchildren.

The report should focus on the indicators and outcome measures recommended by the Task

Force:

1. Length of stay,

2. Number of placements,

3. Re-entry rate and timing, and

4. Exit rates by type of discharge.

The report should provide each entry cohorts size, as well as age, racial/ethnic, and gender

composition. With respect to length of stay, the report should describe how longcohort members

remain in care until their initial discharge, in 90-day increments through 18 months, and in yearly

increments thereafter. In addition, the report should provide quartile distributions that describe

how much time passes before 25 percent, 50 percent, and 75 percent of each cohort is discharged

from placement. Data on the indicators themselves (length of stay, number of placements, etc.)

should be reported by type of initial placement, age at first placement, and permanency planning

goal. The analysis of movements should describe how the probability of movement from one type

of foster care placement is influenced by the prior history of placement.

The annual report should include a separate analysis that explores child safety and

permanency outcomes for specific sub-populations of children. While the foster care system has

broad responsibilities for children and families, certain populations of children are more likely to


25/76


26/76

20

computerized files (reporting lag), and

2. One that measures how many child records were corrected (as opposed to

updated, to reflect new case actions) since the last report. This section will be

accompanied by a description of changes, if any, in the conclusions from previousreports related to corrected records.

Where the evidence indicates persistent data quality problems, they should be analyzed in

the context of current auditing procedures. The Task Force believes that applying existing data

to agency management and outcome monitoring is a necessary if not sufficient condition for

improving data quality over the long term.

Monitoring of ACS and Contract Agency Performance

Although it is important to understand how the foster care system is working as a whole, it is

equally important to understand how the various components of the system are working. To

accomplish this objective, the Task Force recommends that the outcome indicators for this domain

be used to construct monitoring reports for individual agencies, using the indicators described

above. These reports should then be used to evaluate contract agency performance, to monitor

children in the direct care of ACS, and for internal ACS review. These internal reports would

serve as the basis for corrective action plans, future contract negotiations with providers, and the

exclusion of providers based on performance reviews.

With respect to ACS, indicators of system performance such as length of stay and foster

care reentry should be used to evaluate program initiatives. For example, efforts to improve

efficiency and productivity can be understood in part by changes in how long children remain in

care or the time it takes children to pass through service milestones, provided changes in other

outcomes are (i.e., foster care reentry) are unaffected.

Phase Two: Link Information

Like most jurisdictions, the Administration for Childrens Services captures only limited data, so

its database reveals only some of the details. As the analysis of outcome data becomes more

sophisticated, this limited range of information limits what can be known about foster care

outcomes. For example, the CCRS data for New York City does not contain reliable information

on why a child was placed in foster care. As a result, any study of foster care outcomes that relies

solely on CCRS can not use reason for placement as a way to understand outcomes.


27/76

21

Because existing data systems are limited, the Task Force recommends a second phase of

work that focuses on linking information to augment and/or integrate existing data resources.

There are three general strategies for augmenting the data in administrative data systems.

1. Redesign the systems to incorporate a wider array of data about clients, theirfamilies, and the services they receive. Since the Connections Project is a redesign of existing

child tracking systems, this option will not be described in this report.

2. Collect new data, either through case readings or another type of survey

designed to provide more information on a specific subset of the foster care population. In

particular, the Task Force recommends that ACS:

A. conduct periodic studies of the children placed in foster care. Case readings

might be used to develop detailed clinical profiles of the children. These

studies could be developed along with the analysis of placement stability

that will emerge from Phase One.

B. focus, in part, on the sub-populations identified in studies that depend on

existing data resources for example, children who stay in care for short

periods or children most likely to re-enter care.

C. emphasize information on services received (frequency, duration, and

intensity), since so little is known about the relationship between outcomes

and services received.

D. focus, in surveys of satisfaction and well-being such as those described

below, on sub-populations of children defined on the basis of placement

risk, placement duration, re-entry, age, or other characteristics.

3. Link existing sources of computerized data to form a more comprehensive

picture. For example, vital statistics records can be linked electronically to computerized fostercare records. This linkage permits information about a childs status at birth to be used to

understand the risk of placement, as well as the influence that factors such as birthweight have on

placement duration, movements between foster homes, the likelihood of adoption, or the risk of

returning to care. Notwithstanding the critical issue of confidentiality, record linkage represents

an important tool in the effort to better understand foster care outcomes.

Along these lines, the Task Force recommends that the Administration for Childrens


28/76

22

Services develop a data base that links administrative records (CCRS data) with other electronically

stored data. The linked database can be maintained continuously (i.e., updated regularly in whole or

in part) or developed to meet the needs of specialized studies. The data sources that should be linked

include:

A. Census data Census tract data is the best way to understand how

communities impact child welfare outcomes. Moreover, any attempt to

target services by community depends to a large degree on understanding

differences in service utilization from one community to the next.

B. Vital statistics records Over the past decade, nearly 40 percent of the

children entering foster family homes have been infants. Linking birth

records to CCRS would dramatically enrich the available data and allow a

detailed understanding of placement risks and placement outcomes. Birth

records for children admitted to foster care after their first birthday could

also yield benefits.

C. Other service data CCRS is not the only source of child- or family-level

information available to ACS on a routine basis. Facility data, data from

completed risk assessments, and data from the State Central Registry can

also be integrated with CCRS data to enhance understanding of foster care

outcomes. In addition, foster children and their families often receive

services from providers outside the child welfare system. Subject to

confidentiality regulations, whenever and wherever providers track case

histories with computerized tracking systems, those data systems can be

used to build a more detailed profile of clients, the services they receive,

the problems they face, and the impact services have on their lives.


29/76

23

V. Domain Two:

Child and Family Well-Being

Of the three domains the Task Force studied, outcomes related to child and family well-being

proved to be the most challenging. On the one hand, the broad appeal of child and family wellbeing measures reflects the child welfare system deeply held commitment to helping children and

families succeed. On the other hand, first measuring and then interpreting changes in well-being

that can be directly attributed to foster care is a substantial undertaking, especially if the results

are going to be used in a comprehensive system of public and private agency accountability.

The Challenge of Measuring Well-Being

In considering these issues, the Task Force identified three points that bear on the use of well being

indicators as outcome measures for the child welfare system.

1. There is no one set of well-being indicators that fits all the children in foster care.

For any given child, the choice of appropriate indicators involves at least three

considerations:

A. the childs age and developmental status,

B. the length of time he or she has been in placement, and

C. the objectives of the specific program providing care.

2. While foster care placement clearly impacts both the child and the family, isolating

its specific impact on development over the life course is difficult. The task must

be approached with caution.

3. Despite the broad appeal of well-being indicators, the link between well-being and

the mission of the foster care system must be carefully drawn. The foster care

system must pay close attention to child and family well-being indicators in a

systematic performance review. Nevertheless, there is a tendency to expand the list

of social and developmental outcomes beyond those that are central to the mission

of child welfare. The challenge is to construct a reasonable list of well-being

indicators that neither trivializes the role foster care plays in child development nor

overextends expectations.


30/76

24

Bearing these considerations in mind, the Task Force developed the following perspectives

on child and family well-being indicators. With regard to the general category of child well-being,

safety is the indicator of child well-being most central to the mission of child welfare.

Consequently, maltreatment of one form or another serves as the primary indicator of child well

being for all children, regardless of the type of program. Child maltreatment outcomes provide

essential information for judging the quality of substitute care and for determining whether changes

in other outcome measures, such as increases in reunification rates or decreases in length of stay,

represent positive changes for children and families.

Beyond safety, the scope of the foster care systems responsibility for child and family

well-being outcomes is to a degree dependent on how long a child remains in foster care and the

therapeutic expectations of a particular program. For example, when children are reunified with

parents after a brief stay in a traditional foster family home (e.g., less than six months), theabsence of subsequent maltreatment is a more direct measure of success than other measures of

child well-being, such as social/emotional development or cognitive skills. Conversely, longer

lengths of stay mean that the child welfare system must assume broader responsibility for the

social, educational, and developmental well-being of children. Finally, outcomes for children

placed in therapeutic placements should include additional measures of well-being or child

functioning directly related to specific programmatic objectives, such as independent living.

Still, if children and families experience difficulty following reunification but there is nomaltreatment or apparent risk of imminent maltreatment, then those problems should perhaps be

addressed outside of the traditional child welfare system. This conclusion underscores the

importance of relating outcomes to the mission of child welfare, and reminds us that service

continuity and improved child and family well-being often depend on integrated service delivery

within a community context.

In general, then, indicators of child and family well-being other than child maltreatment can

be extremely useful in judging system performance particularly when they are examined in relation

to the goals of child safety, permanence, and least restrictive placement.

1. Several negative indicators of child and family well-being, such as child behavior

problems, domestic violence, and parental drug abuse, are important in this context.

That is, from an outcomes perspective, concern about an issue like parental drug

abuse stems from the increased likelihood of child maltreatment.

2. The wide array of child and family well-being indicators is most relevant for


31/76

25

judging system performance in relation to the length of time a child is in care and

the type of care provided, including the quantity and types of therapeutic services.

Long-term and intensive programs that address child and family functioning should

be expected to affect these outcomes more than short-term and less intensive

services. When the permanency goal is independent living, indicators such as

vocational aptitudes and life skills become important. Still, standards of

performance for these programs also should depend on the length, scope, and

quantity of services.

3. Finally, indicators of child safety other than maltreatment are clearly important

outcomes in and of themselves. Out-of-home care providers have a fundamental

responsibility to address child safety outcomes, such as violence toward self and

others, while children and youths are in care. Increases in these outcomes during

care or following reunification or adoption must raise questions about the adequacy

and effectiveness of services.

Recommended Outcome Measures

Table 2 presents the specific outcome measures and well-being indicators that the Task Force

considered. Child well-being outcomes include indicators of safety and various dimensions of

child functioning, such as physical and emotional health, relationships with family members and

peers, appropriate behavior, cognitive development, cultural identification, and vocational and

independent living skills. Family well-being outcomes include indicators of safety for family

members, parental functioning and education, the problems of parents, and family relationships.

These outcomes are important to examine both during placement and after reunification, adoption,

or independence. During placement, ongoing assessment of child well-being can yield information

that is useful for planning treatment and for determining whether change has occurred as a result

of therapeutic interventions. After placement, measuring the childs safety and functioning can

provide valuable information about the success of planned (or unplanned) permanency

arrangements.


32/76

26

Table 2.Domain Two

Indicators of Child and Family Well-Being

Area of Child and Indicators Issues to ConsiderFamily Well-Being

Child safety 1. Child maltreatment including type, 1. Indicators of abuse and neglecttiming, frequency, and perpetrator.

2. Serious harm or fatality caused bymaltreatment.

3. Violence toward self and/or others.

reflect negative attributes of family

life. Positive indicators emphasize

the absence of abuse and neglect.

Family meets basic needs of 1. Health maintenance including 1. Indicators must reflect what the

children immunizations, well baby visits, and family does on behalf of childrenother developmentallly appropriate health not simply what the family doescare. not do.

2. Nutritous foods.

3. Safe and adequate housing. 2. Housing and income serve as

4. Family promotes childs education.

5. Mental health care.

6. Income levels.

general indicators because of theirclse association with child andfamily well being.

Child emotional and behavioral 1. Violence toward self and/others. 1. Indicators 7, 9, 12 and 13well-being express positive indications of

2. Stays at home (i.e., child doesnt runaway).

3. Acts in sexually appropriate ways.

4. Does not threaten or verbally abuseothers.

5. Is drug and alcohol free.

6. Behavior patterns do not includefiresetting, encopresis, eneuresis, etc.

childs overall adjustment.

2. Expectations and analysis of allthe indicators in this domain musttake the childs age anddevelopmental status, and service

history into account.


33/76

27

7. Maintain control of self.

8. Is free of depressive symptoms.

9. Has positive self-regard grounded incultural/ethnic identification.

10. Psychiatric hospitalizations areavoided.

11. Police contacts ar avoided - noarrests or re-arrests.

12. School attendance is grade and ageappropriate.

13. Amount and quality ofinvolvement with others, especiallypeer relationships that are positive and

age appropriate. Includes relationshipswith other adults, relatives and familyacquaintances.

Childs educational development, 1. School Attendancecompetencies, and achievements

2. Test scores are developmentallyappropriate.

3. Grade point average is appropriatefor age and developmental status.

4. Child talents (music, art, or otherspecial skills).

5. Special accomplishments at home,in school, at the house of worship, orat work (i.e., awards, certificates ofmerit, participation in activities).

Youth self-sufficiency/life skills 1. Has positive vocational skills andwork attitudes.

2. Has employment experiences(depending on age).

3. Understands personal finances(savings and bill payments).

4. Uses social support as neededincluding family, community members,health care, and social services forrecreational and other forms of support.


34/76

28

5. Has earned or is working towardhigh school or equivalent diploma.

Child Health 1. Child is free from harm caused bymaltreatment.

2. Medical history includingimmunizations, well baby visits, andother developmentally appropriatehelath care that is consistent with thechilds health status and age.

3. Teens take steps to avoid pregnancy(including abstinence) and sexuallytransmitted diseases.

Family well-being 1. Parents and other householdmembers are drug and alcohol free (asdetermined through urinalysis ifneeded).

2. Ability of parents to function as

household heads is not impaired for reasons

related to their emotional or psychological

well-being.

Parenting and parent childinteractions 1. Parents protect children frommaltreatment.

2. Parents use appropriate disciplinary

methods.

3. Parents exhibit interest in children,are motivated to carry out parental role,and look forward to reunification.

4. Bonding assessment shows positiveattachments.

5. Parent(s) enjoy interactions withchildren, show respect and affection forchildren and vice versa.


35/76

29


Undertaking a study of child and family well-being costs money. Cost is one of the chief barriers

to using well-being indicators in a system of performance monitoring and program evaluation. As

in the case of client satisfaction indicators, the child welfare system in New York City and otherparts of the country does not systematically process data that describes the well-being of the

children or families involved in foster care. The information burden associated with measures of

well-being has two specific dimensions.

1. Appropriate measurement tools. Sometimes existing instruments may be adapted,

but not before a target population, program goals, and specific developmental

issues have been identified. Target populations have different developmental

needs, and programs have different objectives. The instruments used to measure

outcomes have to reflect this diversity.

2. Reliable baseline data. The cost of developing the basic research capacity to

understand the impact of foster care on well-being has to be considered.

Developing and maintaining data capacity to track child and family well-being over

time will cost more than is now being spent. There will also be a time lag between

the day data collection starts and the day when usable descriptive data are available.

In addition, there will be a second, more substantial period before analysts are able

to use that data to identify how foster care influences child and family development.

Again, since so little baseline data exists, the Task Force proposes a two-phase strategy.

Phase I: Undertake a Study of Child and Family Well-Being

The Task Force recommends that the Administration for Childrens Services complete a systematic

study of child and family well-being within the next 12 months. The studys primary emphasis

should be on child safety. Outcomes should be expanded to include developmentally appropriate

indicators of well-being, such as school performance or readiness for independent living, when the

childs age or other considerations warrant. Still, the scope of the initial assessment should belimited. Initial risk assessment and follow-up risk assessment data that ACS currently gathers

should be factored into the overall design of Phase One.

1. The ACS study should be based on a random sample of foster children stratified by

age, placement history (i.e., prior length of placement), placement type (direct care

and children placed with voluntary agencies), and permanency planning goal. ACS


36/76

30

may wish to add other factors, such as Field Office or Borough. The sample should

specifically include children recently admitted into care, with the expectation

that their case histories will be followed over time. The sample should be large

enough to permit meaningful analysis of well-being.

2. Where appropriate, the primary data collection instrument should be modeled on

the safety assessment that is administered as part of risk assessment. Depending on

the specific sub-population being reviewed, the initial risk assessment should be

supplemented by a follow-up safety assessment. Wherever possible, for purposes

of this limited study of child well-being, later collection of safety data should be

done by someone other the person who completed the original (or most recent)

assessment.

Phase II: Create a Working Group

The most important outcome questions have to do with the impact of foster care on a childs

overall social-emotional development and the well-being of that childs family members. Although

foster care is only one influence in the life of a child, it is important to understand how placement

in foster care influences development over the life course. This question is never easy to answer.

For this reason, the Task Force recommends supplementing systematic studies of child safety with

routine studies of well-being among the children New York Citys foster care programs serve. A

Working Group directed by the Administration for Childrens Services will develop the focus of

these studies. Its issues should include the following:

1. Instrument development:The working group should establish procedures and

criteria for developing instruments to measure well-being. Criteria for selection

include which populations the sample should include; which outcomes, including

safety and other age-appropriate measures, should be targeted; and how frequently

the instruments should be administered.

2. Role of the voluntary agency: In addition to establishing the broad parameters for

a systematic study of well-being outcomes, the working group should also address

the responsibility of contract agencies for studying child well-being outcomes,

including safety.

3. Reporting standards: Finally, the working group should develop reporting standards

that permit comparative analysis of agency performance for purposes of


37/76

31

contracting. Further, the working group should weigh carefully how corrective

action should be related to the reporting standards developed.


38/76

32

VI. Domain Three:

Client Satisfaction

Measures of client satisfaction with services are a valuable source of feedback for policymakers,

program administrators, and direct service personnel. The consumers perspective on the efficacy

of services (e.g., overall satisfaction and satisfaction with specific services) and important

dimensions of service provision (e.g., worker skill and communication issus) may influence

service outcomes such as family reunification. Interest in client satisfaction has grown in recent

years, as models of child welfare practice (including foster care) have increasingly stressed the

importance of including natural parents, extended family members, and foster parents in case

planning and treatment. Parallel developments in business that stress quality assurance standards

and consumer satisfaction have also spurred interest in this rapidly developing area. The

Procurement Policy Board in New York City now requires periodic assessments of client

satisfaction as a condition of receiving a contract.

The Importance of Client Satisfaction Measures

Interest in client satisfaction with services is wholly consistent with the mission of child welfare.

Given the focus of the child welfare system on permanency for children, the perspectives of

children and youths are a valuable and largely underused resource for understanding and evaluating

the effects of services. Second, the ideal of preserving families whenever possible is based on thesignificance of natural parents involvement in the lives of their children. In assessing the child

welfare systems performance, clients (both parents and children) can provide essential information

about whether reasonable efforts are made to preserve and reunify families. For example, youths

and parents can be asked whether services met their needs and if necessary services were available.

Examining their views of various dimensions of the quality of service provision (e.g., cultural

sensitivity, good communication with clients and among professionals, client involvement in

decision making) provides useful information on the adequacy of service provision.

Client satisfaction data can complement system performance outcomes by enriching our

understanding of and ability to interpret a variety of data. For example, if clients voice concerns

about the availability of aftercare services at a time when length of stay is declining, the findings

may be used to strengthen post-discharge services. Alternatively, if length of stay is rising but

clients report high levels of satisfaction with reunification services, it may be desirable to increase

the availability of services with an eye toward reducing time in care. How client satisfaction data


39/76

Task Force recommendations in this section are based in part on the results of a focus group conducted with eight2

parents whose children were or had been in foster care. The results are not intended to be representative of all the

individuals who have feedback to give. Nevertheless, the comments and reactions revealed during the focus groups

indicate how important client satisfaction is to framing an overall sense of outcomes. The results of the focus group are

printed in Appendix X.

33

will be linked to other measures of performance depends on the purpose of the analysis. Clearly,

the clients voice adds a dimension that has been missing from most discussions about how well

the foster care system is working.

Recommended Outcome Measures

The Task Force recommendations that correspond specifically to client satisfaction can be found in

Table II. In reviewing the available literature and talking with clients, four areas within the client

satisfaction domain emerged as particularly relevant to the foster care system in New York:2

1. overall satisfaction,

2. satisfaction with the workers providing services,

3. satisfaction with specific services, and

4. satisfaction with communication.

Within each of these areas, Table 3 also lists specific indicators that form the basis for

conducting focus groups or for constructing instruments that would be used to survey larger

groups of clients.


40/76

34

Table 3.

Domain Three

Client Satisfaction

Area of Client Indicators Issues to Consider

Satisfaction

Overall Satisfaction 1. Global ratings of 1. Information is easy to aggregatesatisfaction across programs, not very specific

2. Overall satisfaction with or refined.specific services (e.g., case 2. Provides general impressions,management, counseling, little specificity about reasonssubstance abuse treatment, for level of overall satisfactionprobation, education, 3. What is relationship of overallrecreation, respite care, satisfaction to systemetc.) performance outcomes?

3. Overall satisfaction with

specific workers4. Would client recommendprogram?

5. Would client return toservices?

Satisfaction with 1. Skills 1. Information should be relevant toWorkers Who 2. Empathy various professionals or easilyProvide Services 3. Availability adapted

Satisfaction with well as workersSpecific Services 1. Safety of child

4. Honesty in communication 2. Should specify which worker5. Respect for clients 3. May be relevant to overall6. Level of trust satisfaction and to system7. Cultural sensitivity performance outcomes

including language spoken 4. Should include foster parents as

2. Visitation opportunities 1. Services usually refers to specific3. Availability of services programs, but satisfaction with4. Availability of client advocacy case management services, a

and support groups childs placement, and visitation5. Waiting lists/delays are also important6. Aspects of service delivery 2. Aspects of service delivery can7. Did services meet needs? be practice principles or ideals8. Length and amount of (e.g., family centeredness: Allen

services and Petr, 1995)

9. Sorroundings/facility 3. May be related to overall10. Satisfaction with placement satisfaction and to system11. Confidentiality performance outcomes12. Satisfaction with outcome13. Cultural sensitivity


41/76

34

including language14. Sensitivity to religious

preferences

Satisfaction with 1. Informed of events 1. May be related to overall

Communication and 2. Informed of rights

Involement 3. Involved in decision making

4. Time to communicate with

lawyers

5. Inter-agency collaboration

6. Adequate opportunity to be

heard from in court

satisfaction and to system

performance outcomes


Few child welfare agencies today systematically gather data on client satisfaction. Moreover, few

client satisfaction surveys have been field-tested with foster children and their parents. The

instruments that do exist have been administered by individual programs, but not on a system-wide

basis. For all these reasons, the cost of developing assessment tools and conducting regular

surveys of client satisfaction-the information burden-will be considerably higher than for otherrecommended measures. While the instruments used to assess satisfaction can be quite short and

easy to administer, data collection, data entry, and analysis across all cases present a secondary,

but nevertheless substantial cost. So that measures of client satisfaction become a regular form of

feedback within the child welfare system, the Task Force makes the following recommendations.

Phase I: Implement Guidelines and Surveys

To meet the short-term need for information from clients, the Task Force recommends two steps

that ACS could carry out as Phase One of a comprehensive plan to gather client satisfaction data.

1. Implement guidelines for uniform questions that would be compatible with the

requirements of the Procurement Policy Board.

2. Implement a targeted survey of client satisfaction modeled after the rapid

assessment technologies used in the health and behavioral health care sectors.

Initially, these assessments should involve small, well-targeted samples of children.

Rapid assessment technology is an approach to client feedback that emphasizes

short, easily answered questionnaires. Data can be quickly entered and analyzed.


42/76

36

Development of the rapid assessment technology should be both guided and complemented

by feedback from focus groups of children and parents. Feedback from clients should be sought

in four basic areas:

1. Demographic and service information,

2. Overall satisfaction,

3. Child safety, and

4. Progress toward permanency.

Instruments should be administered to a sample of children stratified by age, permanency

planning goal (or time in the system), and type of placement. The sample should be large enough

to show meaningful differences in satisfaction. If children are too young to participate directly,

parents should be asked to complete the assessment. When identifying potential samples, special

consideration should be given to clients taking part in innovative programs, so client satisfaction

data can be included in the program evaluation.

Until a more detailed alternative can be developed, the rapid assessment should be

administered to children and their families in panels of roughly the same size. Children who

remain in care would remain part of the sample until they are discharged. A sample of children and

families should be interviewed at the time of discharge. Where possible, and subject to

confidentiality standards, data gathered as part of the assessment should be linked to administrative

data from other sources, including data from the Child Care Review Service (CCRS) and the State

Central Registry (SCR). This will allow the analysis of client satisfaction to reflect the childs

service history and other factors.

Phase One should include three additional steps

1. Encourage providers who are already collecting client satisfaction data to use some

common questionnaire items. These common questions might measure overall

satisfaction and satisfaction with specific services and/or workers;

2. Require both public child welfare agencies and private providers to annually survey

small, representative samples of clients by program (e.g., residential, foster

care, family reunification) or group (natural parents, youth, children) in order to

collect baseline data systemwide;

3. Conduct periodic focus groups in order to hear the views of clients in different

programs; to gather input on program decisions, staff training, and program


43/76

37

development; and to amplify the findings from other studies of system performance

and client well-being.

Phase II: Establish an Advisory Panel

ACS should establish an Advisory Panel to develop a long-range strategy for gathering client

satisfaction input. The Advisory Panel could begin by developing instruments for Phase One, but

its primary purpose would be to develop a long-range plan that addresses costs and procedures

for measuring client satisfaction across representative populations and their service providers.

The Advisory Panel ACS convenes should reach agreement on the following issues:

1. Outcomes related to overall parent and child satisfaction, satisfaction with specific

services, satisfaction with workers, and satisfaction with communication. Detailed

descriptions of these areas and specific indicators associated with each area are

provided in Table 3, above.

2. Recommendationson the continued use of rapid assessment technology and theadaptation of more detailed instruments for in-depth analysis of sub-populations.

3. Standards for assessment that identify the target populations and the frequency

of assessment.

4. Plans to bring recipients of services other than foster care into the satisfaction

review, including but not limited to recipients of preventive services, adoption

services, and child protective services.

5. Provisions for including a broader array of clients, so parents, foster parents,

adoptive parents, and provider organizations have a regular means of providingfeedback.

6. A plan that describes how the collected information will be used and how it will

be disseminated.

7. A statement that defines the responsibilities of public and private agencies for

assessing client satisfaction. In particular, the advisory panel should consider

whether the tracking of client satisfaction data can be integrated with the

Connections Project, and if so, how.


44/76

38

VII. Summary

Properly constructed and used, outcomes and outcomes measures can have a dramatic

impact on the quality of child welfare services available to New York Citys children. Yet,

identifying those outcomes is but a first step in the long process of improving the child welfare

system. Along with outcomes, the system itself must maintain a separate but equally important

focus on the both the structure and processes that define child welfare service delivery. In the end,

it is the combination of process and outcomes that are inextricably linked to highly effective

services.

Members of the Comptrollers Task Force readily acknowledge that this report is a work

in progress. The integration of outcome data into the delivery of services depends on yet another

level of specificity and detail that was beyond the scope of this report. Moreover, the use of

outcome data will place new demands on the individuals who work with families each and every

day. Their need for new competencies must be acknowledged if outcome data is to be collected

reliably and, more importantly, used productively. Finally, no one should under estimate thedifficult task of analyzing and interpreting outcome data. If the data are to be used to understand

what placement in foster care means for children and their families, policy makers among others

must recognize that complex human problems often times command complex answers.

In short, this Report of the Comptrollers Task Force represents an attempt to move the

state of the art but one small step forward in the hope that the questions raised and the answers

provided will stimulate the important work that remains to be done.


45/76


46/76

1

Methodological and Conceptual Issues that Arise

In Conducting Research on Child and Family Well-Being

Like client satisfaction, measures child and family well-being pose important methodological andconceptual hurdles that must be addressed before indicators of well-being can be introduced into

a system of routine feedback and performance review. As an orientation to these issues, the

following review of the literature is intended to serve as a guide for those interested in

understanding how others have used these measures to understand the impact of the child welfare

system on child and family well-being.

Connecting System Peiformance Indicators and Child and Family Well-Being. There have been

some efforts to examine the relationship between child and family functioning and systemperformance such as length of stay and foster reentry. The most developed discussion of this issue

occurs in evaluations of Wraparound services for behaviorally challenged youth and their families.

Researchers recognized the need for socially valid indicators of service effectiveness to

complement clinical measures of functioning, which often yield fairly abstract findings that are

difficult for people outside of the professional research community to interpret (Bruns et al.,

1996). Furthermore, measures of clinical adjustment, while worthwhile, did not address issues of

accountability and cost-related concerns typically encountered by public agencies. In response,

they developed a set of behavioral indicators that providers, advocates, and community residents

thought would be most likely to lead to placement of the youth, resulting in a set of indicators are

quite simple, reliable, and easily interpretable. Similarly, some highly regarded substitute careprograms, such as the independent living programs at The Children's Village, utilize both

traditional clinical assessments of functioning (e.g., locus of control) and concrete indicators of

functioning (e.g., work experiences, school success--which can be seen as system performance

outcomes for these programs) in examining outcomes.

Data Collection Strategies. Data on child and f amily well-being can be collected from parents,youth, and professionals--usually professionals who are working with children and/or families or

who are conducting CPS investigations. Data collection strategies designed to gather quantitative

data on child and family well-being include risk and family assessment instruments, child behavior

and functioning instruments, and other research tools, such as rapid assessment instruments, goal


47/76

2

attainment scaling, and global assessments of functioning. The following discussion is intended to

illustrate some potentially useful strategies and is not a comprehensive list or review of such

measurement instruments.

There are risk assessment and family assessment instruments that are specifically designed for use

in child welfare. These instruments are intended to be applicable to all child welfare cases and they

are of direct relevance to understanding system performance because they assess the factors that

influence the likelihood of subsequent maltreatment and whether children can be protected.

Assessment instruments are used to guide case planning and decision making and sometimes to set

workload requirements (based on the level of risk). These assessments are conducted during both

the initial phase of intervention or investigation and, in most approaches to risk assessment, also at

one or more subsequent stages in a case.

A wide variety of risk factors are examined by various risk and family assessment instruments,

including dimensions of parenting, history of maltreatment, parent-child interactions, individual

functioning of parents (e.g., substance abuse, mental illness), family functioning (e.g., domestic

violence), family support, and child behavior.

Pecora et al. (1992) describe four approaches to risk assessment. 1) Matrix approaches in which

various risk factors, usually identified by clinicians, are rated on a scale that is based on

descriptions of levels of severity for each factor. Many states, including New York, Illinois, and

Washington, use such matrices, which are the only child and family well-being measurementinstruments that are sometimes administered system-wide. These instruments are relatively easy

to use and do not represent an additional information burden in terms of data collection. 2) The

empirical predictoi;s method attempts to determine the specific factors that are most predictive of

subsequent maltreatment (e.g., Baird, 1988; Johnson and L'Espearance, 1984). Such analyses

result in a relatively small number of fairly simple and objective factors (e.g., prior involvement

with the child welfare system) that predict subsequent maltreatment in 70-83% of cases. The

factors used in the matrix approaches have also been examined for their predictive validity in a

small number of studies and they yield similar predictive accuracy (e.g., Marks and McDonald,

1989). 3) Family assessment scales (e.g., McCroskey and Nelson, 1989; Magura and Moses,

1986; Magura, Moses, and Jones, 1987) are not technically designed to predict risk, but they aredesigned to assess parent, child, family, and household functioning in child welfare cases. These

instruments use behaviorally anchored scales and provide more refined measurement of

functioning than risk matrices approaches, but they are also more time consuming. The Child Well-

Being and Family Risk scales (Magura and Moses, 1986; Magura, Moses, and Jones, 1987)) have


48/76

3

been shown to b

hevesi, foster care outcomes and performance standards, 1997

Documents