Exemplary Evaluations

Lessons Needing Deeper Learning: A Reaction to Bickman

JOHN LANDSVERK

INTRODUCTION

Dr. Bickman has laid out nine lessons that evaluators might learn from the Fort Bragg Evaluation. My comments will address three of these lessons (numbered here to correspond with Bickman’s numbering), as well as one lesson I have added. I will discuss each lesson from the perspective of research and program planning for child and adolescent mental health services.

Lesson 2: Develop Program Theory

The Fort Bragg Evaluation is notable in the field of child and adolescent mental health

services research because it systematically and directly tested the single most influential framework for reforming the delivery of children’s mental health services. This framework, which has become known as a “system of care model,” was spurred by Jane Knitzer’s publication of Unclaimed Children in 1982, conceptualized by Stroul and Friedman in their 1986 publication of A System of Care for Children and Youth with Severe Emotional

Disturbances, and developed as a federal public policy initiative in the Child and Adolescent Service System Program (CASSP). For convenience, the “system of care model” will be referred to hereafter as the “integrated system.”

To date, the Fort Bragg Evaluation is by far the most comprehensive and rigorous attempt to test this dominant framework. The development of a program logic model was crucial to this evaluation, as noted by Dr. Bickman. While the program initiatives noted

above had set out general principles for the integrated system, considerable specification of the logic underlying that system was required before an empirical test of it could be performed. Although this study has received careful scrutiny in several national forums,

John Lnndsverk l Director of the Center for Research on Child and Adolescent Mental Health Services, Children’s Hospital,

San Diego.

Evaluation Practice, Vol. 17, No. I, 1996, pp. 75-77. Copyright @ 1996 by JAI Press, Inc.

ISSN: 0886-1633 All rights of reproduction in any form reserved.

75

76 EVALUATION PRACTICE, 17(l), 1996

no-one has suggested that the model of program logic used by the evaluators is not faithful to the general program principles in the program initiatives. Had the evaluators’ program theory been flawed, the study would have been quickly discredited. This underscores the importance of developing sound program theory and the critical role that such theory serves for program evaluators. Evaluators and program planners are sure to find the program model developed for the Fort Bragg Evaluation useful in future research studies and program development in child and adolescent mental health services.

Lesson 5: Measure Cost of the Program

Another lesson listed by Dr. Bickman is the need to measure the costs of the program, but he doesn’t emphasize this lesson enough. The need to consider costs is perhaps even more critical when the program being evaluated is driven by public policy and public resource commitments. Usually, evaluators consider cost analyses to be most important when program effectiveness is demonstrated. In fact, this appears to be Dr. Bickman’s position when he suggests that “additional funds should be more readily available to conduct a cost analysis of an effective program.” However, the Fort Bragg Evaluation indicates that measuring costs can be very important even when program effectiveness is not demonstrated in ultimate outcomes such as better client functioning. In this study, the integrated system and the usual delivery system were found to yield the same decrease in client symptomatology and the same increase in client functioning (ultimate outcomes). This finding would not be so troubelsome to advocates of the integrated system if the unit costs had been the same or lower under that model. In fact, even a finding that costs of the integrated system and the usual system were equivalent would have been a very positive finding, given the strong positive impact of the integrated system on service access, timely service delivery, less use of restrictive care, and greater client satisfaction. However, the integrated system has long been assumed to save costs because of projected decreases in use of restrictive care settings (e.g., psychiatric hospitalization) and increases in use of community based services (e.g., outpatient services). Therefore, the sharply increased unit costs, in the context of lowered use of restrictive levels of care, suggests problematic elements in the overall integrated system model. The lesson for evaluators is the potential importance of program cost analysis, even when program effectiveness is not borne out in ultimate outcomes.

Lesson 8: Deal Directly with True Believers

In this lesson, Dr. Bickman suggests to evaluators that they should “deal directly with true believers.“This is based on the critical interplay that has occurred between researchers and program advocates who were strongly linked to the integrated system and the evaluators who carried out the Fort Bragg Evaluation. Dr. Bickman should be lauded for his active participation in dialogue with other researchers and advocates. Also exemplary is the timely and comprehensive publication of results from the study which ha.s provided a full public disclosure of methods and findings. Both of these evaluator behaviors provide role models for all evaluators who study programs which are undergirded by strong public policy commitments.

However. Dr. Bickman’s discussion also suggests an additional lesson for evaluators of programs with a strong advocacy base. namely, the potential for the evaluator to become

Exemplary Evaluation 77

overly engaged in the policy dispute arising from unexpected negative findings. That this may have occurred is suggested by Dr. Bickman’s characterization of the advocates as “true believers” and his suggestion that “critics have tried to ignore the study.” The use of Eric Hoffer’s memorable term implies that the criticism of the Fort Bragg Evaluation is founded in a fanatic’s unalterable position. My observations of the sometimes spirited dialogue and debates over the findings from the Fort Bragg Evaluation would suggest rather that the advocates and researchers linked to the integrated system have raised important questions which needed to be considered when interpreting the results of this study. Dr. Bickman has certainly addressed those questions. However, it is likely that considerable time will still be needed by the field of children’s mental health services to

understand the full implications of the unexpected dual findings of (1) no differences in ultimate outcomes; and (2) sharply increased costs under the integrated system.

Dr. Bickman’s unfortunate use of the “true believer” label suggests the danger for

evaluators in becoming overly engaged in the debate about the public policy implications of their study.

This potential for over engagement is also suggested by Dr. Bickman’s assertion that “critics have tried to ignore the study, maybe in the hope that it would just go away.” This study has received enormous attention over the past five years, with numerous lengthy discussions in national conferences of the study’s findings and implications. The one example given where evaluators were not invited to participate in a panel discussion is more than balanced by numerous examples of widespread interest in this study and a sincere struggling with the meaning of its findings. Again, the language used by Dr. Bickman suggests the potential for evaluators to become “embroiled” in the public discussion of their study findings, especially when the findings are contrary to the expectations of program advocates who have been operating in a vacuum of research findings.

Final Lesson

In my opinion, the most important lesson that can be learned from the Fort Bragg Evaluation is that large scale system change can be tested rigorously, if sufficient resources and evaluation expertise are available. The Fort Bragg Evaluation demonstrated that a human service model based on general principles can be fully specified and tested in difficult field conditions. This study also demonstrated the importance of using considerable evaluation resources to determine whether the program was implemented with fidelity and whether the program services were delivered with sufficient quality, as expected by the program model. The Fort Bragg Evaluation stands as a model of evaluation that should be studied by all evaluators of large scale system change.

NOTE

I. Dr. John Landsverk is the Director of the “Center for Research on Child and Adolescent Mental Health Services” at Children’s Hospital-San Diego. This is one of two research centers funded by the National Institute of Mental Health that focus on mental health services for children and adolescents.