Staff Selection and Support Materials to Support Communities

Implementing High Fidelity Wraparound

Vroon VanDenBerg, LLP May 2008

© Vroon VanDenBerg LLP 2008

Model for Implementing High Fidelity Wraparound Wraparound emerged in the 1980s as a value-driven approach to providing community-based care for youth with severe emotional challenges and their families. Only recently has there been agreement on the primary elements of the model1 2, standardized methods to measure fidelity to the process3 and a specified model of service delivery4. Such steps are important because research using a variety of measures and methodologies has demonstrated that the quality or fidelity of wraparound varies greatly5 and that the fidelity of the process directly correlates with the outcomes for children and families6 7. Research has shown that high fidelity wraparound has been effective in engaging and empowering families, producing good outcomes for children and families while reducing high cost services. Research has also shown that low fidelity “wraparound” has not engaged families in the process, has produced poorer outcomes for children and families and often increases the cost of services8 9 10. One of the challenges for implementing high fidelity wraparound has been moving from pilot projects to wraparound implemented within and across community-based programs. Fidelity assessments have consistently documented that the wraparound being offered through these programs does not reach or sustain the fidelity levels associated with the best outcomes for children and families. What is needed is a model of implementation that supports the development and sustaining of high fidelity wraparound. Based on the work of Fixen, Naoom, Blasé, Freidman, and Wallace11, we have developed an implementation model that describes the primary components of a system of supports. The figure below shows these components.

Community Context and Readiness

StaffSelection

Training

Supervision and

Coaching

Performance Management

ProgramEvaluation

OrganizationalSupports

© Vroon VanDenBerg LLP 2008

The first component of the implementation model is community readiness. Our experience in over 50 communities across North America has demonstrated the need for communities to get ready to provide wraparound before selecting or training staff or enrolling youth and families. The process of the readiness assessment is covered in another manual12.

Staff Selection The second component is the selection of staff to provide wraparound. One of the significant challenges in providing high fidelity wraparound is getting staff to fidelity levels and then maintaining them in their jobs. Early implementation efforts found that it took an average of 9 months for facilitators to reach high levels of fidelity13 and programs report turnover in positions close to 100% per year14. The implementation goals are to reduce the time to fidelity and to decrease the rate of turnover. An examination of exit interviews in three programs found that many of the people hired to do wraparound did not understand the expectations of the job and that many did not feel well suited to the demands. In addition, exit interviews with supervisors found that effective facilitators had specific personality characteristics which made them more effective and the job more satisfying. Staff selection strategies that help potential staff have a better understanding of job requirements and better identify staff with the effective personality characteristics should improve job satisfaction and decrease turnover. In several studies implementing these strategies, programs provided clear descriptions of what was expected of staff and behavioral rehearsals as part of the interview process to look for the personality characteristics that seem a good match for the job. Using these strategies resulted in decreases in turnover from 60 to 19% of staff after one year in three programs15.

41.2

81.2

58.8

18.8

0

20

40

60

80

100

Regular Hiring Enhanced Hiring

Perc

ent o

f Hire

s at

One

Yea

r

LostRetained

© Vroon VanDenBerg LLP 2008

In the next section of this manual there are lists of characteristics of good facilitators and family support partners, sample descriptions of the requirements of wraparound to use for informing staff, and behavioral rehearsals to better identify staff with the “right stuff”.

Training and Coaching The next components in the implementation model are training and coaching of staff. Training is defined as workshop or e-learning sessions that teach wraparound through a variety of strategies and home-work assignments. Coaching is the process of teaching someone to do wraparound by showing them the process, consulting with them on how to use the process, and helping them discover the craft knowledge to do it well. Training transfer describes how much impact training has on wraparound fidelity, but even the best training is not sufficient to produce high fidelity wraparound. Coaching is necessary to complete the development of the skills to provide high fidelity wraparound and on-going coaching is necessary to sustain it. The following study helps to visualize the impact of training and coaching. This study compared the level of fidelity following training and then the level following coaching for 18 facilitators in a community mental health system16. The graph below shows the impact of training and coaching on the fidelity of wraparound as measured by the Wraparound Fidelity Index (WFI17). High fidelity wraparound is defined by a score of 85% or higher on the WFI. The results show that training moved staff about one third of the way between baseline and expected high fidelity levels as scored on the Wraparound Fidelity Index. With on going coaching support the facilitators in this study were able to achieve high fidelity scores.

72%

86%

64%

50%

55%

60%

65%

70%

75%

80%

85%

90%

Pre Training Training Only Training and Coaching

Phase in Wraparound Fidelity Process

Aver

age

Fide

lity

Scor

e

© Vroon VanDenBerg LLP 2008

The second study was done in Nevada as part of a controlled evaluation of the impact of wraparound compared to practice as usual for 67 children and youth in the child welfare system18. A multiple baseline design with initial training followed by field based coaching was used to assess the impact of fidelity on outcomes for the youth. The data presented is from the Child and Adolescent Functional Assessment Scale (CAFAS19). The CAFAS measures the functional impact of behavioral health symptoms across eight live domains. Lower CAFAS scores show less impairment. The results show that the fidelity of wraparound is directly related to outcomes and that fidelity and outcomes can be improved with coaching.

-70-60-50-40-30-20-10

010

Trad MH Services High FidelityWraparound

Low Fidelity thenCoaching

Chn

age

In C

AFA

S Sc

ores

6 months 18 months

Coaching is a time intensive and expensive part of the implementation model. Improving training transfer can decrease the time to fidelity and amount of coaching required. Two field evaluations across six groups of trainees examined strategies to improve the impact of training. Each of these studies built on fidelity data gathered on previous students soon after completing training. Vroon VanDenBerg (VVDB) has taken the phases and activities of wraparound as defined by the National Wraparound Initiative20 and defined specific skill sets for each activity. VVDB then developed coaching tools to measure fidelity to these skill sets.21 VVDB coaches then use these tools to monitor the progress of new facilitators to fidelity. These two studies looked at the impact of training describing the activity, training focusing on the details of the skill sets, and practice with the skill sets during the training (behavioral rehearsal).

© Vroon VanDenBerg LLP 2008

50.055.060.065.070.075.080.085.090.095.0

Skill Set1

Skill Set2

Skill Set3

OverallScore

Aver

age

Sco

re

Training Skill Sets Behavioral Rehearsals

The results found that some of the skill sets had good transfer when training gave specific requirements, for some defining the skill sets were sufficient to reach fidelity and for others behavioral rehearsal was needed to get good transfer. Based on these ongoing field studies the curriculum has been periodically updated and refined and now results in almost twice as much transfer.

Supervision and Coaching The fifth component in the implementation model is supervision. Research has shown that strengths-based supervision that focuses on staff performance and proactively monitors work with youth and families results in better outcomes for youth and families and better satisfaction for employees22. In another field evaluation the impact of supervision type and frequency was examined on the time for staff to reach fidelity. This study compared the use of supervision as needed (supervision times were not regularly scheduled and were inconsistently provided at a low frequency focusing primarily on crisis situations), to low frequency scheduled supervision (less than three times a month), to proactive and frequent supervision (at least three individual and two group sessions per month). The results found that proactive coaching and supervision can cut the time to fidelity by almost 50%.

© Vroon VanDenBerg LLP 2008

02468

101214

Inco

nsiste

n t

Low Fre

q

Proacti

ve

Mon

ths

to F

idel

iInconsistent super/coach is more crisis and reactiveLow frequency super/coach meets less than weekly and rarely does in-vivo sessionsProactive supervision implements plans for professional development

In the final field evaluation the use of specific skill based monitoring for coaching was compared to more principle focused supervision. The use of skill sets and frequent credentialing decreased the time to fidelity by over 40%.

General coaching focuses more on the priorities of the mentor or staffSkill set coaching focuses on skill sets but does not consistently measure progressCertification coaching does both

02468

1012

Cer

tifi

cati

on

Ski

ll S

ets

Gen

eral

Co

ach

ing

Mo

nth

s to

Fid

eli

1 Burns, B. J., Goldman, S. K., Faw, L., & Burchard, J. D. (1999). The wraparound evidence base. In B. J. Burns & S. K. Goldman (Eds.), Systems of Care: Promising Practices in Children's Mental Health, 1998 Series (Vol. IV). Washington, DC: Center for Effective Collaboration and Practice, American Institutes for Research.

© Vroon VanDenBerg LLP 2008

© Vroon VanDenBerg LLP 2008

2 Suter, J.C., Bruns, E.J. (2008). A Narrative Review of Wraparound Outcome Studies. 3 Bruns, E. J., Suter, J.C. Force, M. M., & Burchard, J. D. (2005). Adherence to wraparound principles and association with outcomes. Journal of Child and Family Studies, 14, 521-534 4 Walker, J.S. & Bruns, E.J. (2006). Building on practice-based evidence: Using expert perspectives to define the wraparound process. Psychiatric Services, 57, 1579-1585. 5 Walker, J. S., Koroloff, N., & Schutte, K. (2003). Implementing high-quality collaborative individualized service/support planning: Necessary conditions. Portland, OR: Portland State University, Research and Training Center on Family Support and Children's Mental Health. 6 Rast, J. O’Day, K., and Rider, F. (2005). A Post Hoc Comparison of Child and Family Outcomes to Fidelity of the Wraparound Process for Project MATCH. In Newman, C., Liberton, C.J., Kutash, K. & Freidman, R.M, (Eds.), The 17th Annual Research Conference Proceedings, A System of Care for Children’s Mental Health: Expanding the Research Base. Tampa: University of South Florida, The Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health. 7 Rast, J., Peterson, C., Earnest, L. & Mears, S. (2004). Service Process as a Determinant of Treatment Effect – the Importance of Fidelity. In Newman, C., Liberton, C.J., Kutash, K. & Freidman, R.M, (Eds.), The 16th Annual Research Conference Proceedings, A System of Care for Children’s Mental Health: Expanding the Research Base. Tampa: University of South Florida, The Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health. 8 Rast, J., Bruns, E. J., Brown, E. C., & Peterson, C. R. (2007). Outcomes of the wraparound process for children involved in the child welfare system: Results of a matched comparison study. Unpublished manuscript. 9 Bruns, E. J., Suter, J.C. Force, M. M., & Burchard, J. D. (2005). Adherence to wraparound principles and association with outcomes. Journal of Child and Family Studies, 14, 521-534. 10 Rast, J., Vetter, J., Poplin, E. & Anderson, N. (2008, February). Wraparound in Oklahoma: Preliminary results of a randomized study. Paper presented at the 21st Annual Research Conference: A System of Care for Children's Mental Health, Louis de la Parte Florida Mental Health Institute, Tampa, FL. 11 Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research 12 Rast, J. (2003). Quality Management and Continuous Quality Improvement. Processes to support wraparound and System of Care Development. Ohio Department of Mental Health and Substance Abuse. Columbus, Ohio

© Vroon VanDenBerg LLP 2008

13 Rast, J. & VanDenBerg, J. (2006, February). High Fidelity Wraparound – Research and Issues for Implementation. Paper presented at the19th annual Research Conference: A System of Care for Children's Mental Health, Louis de la Parte Florida Mental Health Institute, Tampa, FL. 14 Rast, J. & VanDenBerg, J. (2007, February). Using Evaluation to Implement and Sustain the Fidelity of Wraparound – Lesson Learned and Issues for Implementation. Paper presented at the 20th Annual Research Conference: A System of Care for Children's Mental Health, Louis de la Parte Florida Mental Health Institute, Tampa, FL. 15 Rast, J. & VanDenBerg, J. (2008, February). Impact of Training and Coaching on Wraparound Fidelity: Results of Field Evaluations and Practice Research. Paper presented at the 21st Annual Research Conference: A System of Care for Children's Mental Health, Louis de la Parte Florida Mental Health Institute, Tampa, FL. 16 Rast, J. & VanDenBerg, J. (2008, February). Impact of Training and Coaching on Wraparound Fidelity: Results of Field Evaluations and Practice Research. Paper presented at the 21st Annual Research Conference: A System of Care for Children's Mental Health, Louis de la Parte Florida Mental Health Institute, Tampa, FL. 17 Suter, J.C., Burchard, J.D., Bruns, E.J., Force, M.D., & Mehrtens, K. (2002). User’s Manual to the Wraparound Fidelity Index 3.0. Burlington, VT: University of Vermont Department of Psychology. 18 Rast, J. Peterson, C., Earnest, L., and Mears, S. (2003). Treatment Effectiveness: Comparing Service Process Between Wraparound Process and Traditional Mental Health Services. 16th Annual Research Conference Proceedings. Tampa, Florida. 19 Hodges, K., & Wong, M.M. (1996). Psychometric characteristics of a multidimensional measure to assess impairment: The Child and Adolescent Functional Assessment Scale. Journal of Child and Family Studies, 5, 445-467 20 Walker, J.S., Bruns, E.J., VanDenBerg, J.D., Rast, J., Osher, T.W., Miles, P., Adams, J., & National Wraparound Initiative Advisory Group (2004). Phases and activities of the wraparound process. Portland, OR: National Wraparound Initiative, Research and Training Center on Family Support and Children’s Mental Health, Portland State University. 21 Rast, J., VanDenBerg, J., and Schick, C. (2007). Wraparound Facilitator Credentialing Manual. Parker, Colorado, Vroon VanDenBerg, LLP 22 Rast, J. & VanDenBerg, J. (2007, February). Using Evaluation to Implement and Sustain the Fidelity of Wraparound – Lesson Learned and Issues for Implementation. Paper presented at the 20th Annual Research Conference: A System of Care for Children's Mental Health, Louis de la Parte Florida Mental Health Institute, Tampa, FL.

Wraparound Fidelity Index 4.0 Summary Report

Condensed State Report

Data Collection Period July 2007—December 2007 Prepared by Innovations Institute, University of Maryland With

Consultation from the Wraparound Evaluation and Research Team, University of Washington

Innovations Institute

Division of Child and Adolescent Psychiatry University of Maryland

737 W. Lombard St., 4th floor

2

Purpose The purpose of this evaluation is to determine the extent to which the services and supports that are being received by children, youth, and families enrolled in care management within Maryland adhere to the foundational principles of the Wraparound process. Additionally, the report will guide: continuous quality improvement by the Wraparound provider; coaching by Innovations Institute; monitoring of quality of the Wraparound service delivery by the Local Management Boards (LMBs); and performance measures reported to the State of Maryland. Respondents

(Total Eligible Teams N=88) Respondent Consent

to Contact

Completed Interviews

Decline Could not reach

Completion Rate*

Care Coordinator

88 76 6 6 81%

Caregiver 57 23 13 21

52%

Youth 43 13 7 23

36%

Team Member

53 29 6 18 60%

Summary

Over the past six months Wraparound fidelity monitoring has reflected high quality and good overall adherence to the process and principles of Wraparound. It should be noted that this is the first reporting period for the Wicomico County jurisdiction and that the St. Mary’s jurisdiction data will be included in the next reporting cycle ending in June, 2008. At this time in implementation there is little variability by jurisdiction for the combined Total WFI and Element scores. There was however, significant variance found when examining WFI scores by respondent type and jurisdiction. This may suggest that the principles of Wraparound may not be consistently understood by all members of the Wraparound team and the level of understanding by the different respondent types varies by jurisdiction.

When examining the ten WFI element scores, Cultural Competence, Family Voice and Choice, Strengths Based and Collaborative all reached 90% fidelity or above indicating strong adherence by Wraparound teams to these principles. Two elements however, fell below adequate fidelity 1) Individualized and 2) Natural Supports. When examined further, one specific item, which fell below 1.0 on the fidelity scale pulled the Individualized element score below 70%, “Does the family’s Wraparound plan include mostly professional services.” This concept of building in non-professional community based services into Wraparound plans is consistent with the idea of increasing natural supports to the family, which is demonstrated by actively seeking out and encouraging the full participation of team members drawn from family members’ networks of interpersonal and community relationships. The result of increasing natural supports will likely be greater opportunity to implement non-professional, individualized and community-based activities for the youth. It is also important to note that although the elements of Persistent, Outcomes-Based, Team Based and Community Based are all currently reflecting adequate WFI scores, these are areas that also can be strengthened and improved.

3

Important caution in interpreting the findings from this study should be taken because of restrictions on completing the interviews with all families that were identified for fidelity monitoring. The improvement in fidelity monitoring, while having confines, most likely is the result of stability in leadership within the CME, staffing stability and increasing consistency between the supervision experience, training and coaching. Crucial challenges for all the jurisdictions involved in Wraparound lies in developing individualized plans and finding natural supports to serve on child and family teams. Recruiting such supports is often difficult, but the presence of natural supports on the team is a critical part of sustaining a functioning team and also may provide perspectives that lead to greater individualization of plans to better meet youth and family needs. See Figure 1. for an example of Element Scores broken down by respondent type. Strengths • Overall Wraparound fidelity across funding, respondents, and elements is averaging between

“good” and “superior.” • The elements of cultural competency; family voice and choice; collaborative; and strengths based

are reporting superior fidelity across all respondents. • Caregivers and team members are reporting on the overall professionalism and commitment of

Care Coordinators. In particular, caregivers report that the success of the program is in its ability to broaden access to services and supports.

Areas of Improvement • The elements of the Wraparound process: persistent, team based, community based and outcome

based, show adequate fidelity at this time, but are areas that could be improved upon in order to reach the goal of >85% fidelity.

• The elements of Natural Supports and Individualized plans are showing borderline fidelity and need to be specifically addressed through training and coaching.

• Care coordinators repeatedly mentioned that engaging families in the Wraparound process was a struggle they had experienced with some families. Therefore, enhancing skills related to family engagement will be important for improved fidelity.

Recommendations • Innovations Institute, MD Choices, New Transitions and LMBs partner to provide more training

for Child Serving Agencies and providers within all jurisdictions providing Wraparound with an emphasis on areas of improvements.

• Care Management Entities (CMEs) take more time to engage family and other team members in the process to increase awareness of elements and components of high fidelity Wraparound that may or may not already be taking place. .

• Innovations Institutes coaches will regularly attend CFT meetings and complete document review measures in each jurisdiction.

• Innovations and each of the jurisdictions will work together to increase the number of Caregiver, Youth and Team Member respondents for future reports.

• Innovations will host discussions with Jurisdictions to review WFAS implementation lessons learned and proposed data collection models for Fiscal Year 2009.

Figure 1. Comparison of Care Coordinators (74), Caregivers (24), Team Members (26), Youth (13) Element Percentages Description. The following graph compares each element scores by the four respondent type across funding sources combined. However, it must be noted that the receipt of data from each member of a team was rare, preventing direct comparisons between the respondent types. The following data are not comparing caregivers and care coordinators on the same team delivering services. Therefore, the following graph should be used only to assess where there is general agreement and disagreement between different respondents about the same type of services.

79 80

95

64

8193

50

7991

68

96

65

89

65

95

7281

91

72

94

0102030405060708090

100

Culturally Competent Individualized Strengths Based Persistent Outcomes Based

Perc

ent F

idel

ity Care CoordinatorCaregiverTeam MemberYouth

908492

77

62

7786

7364

7680 84 7688 87

54

757668

82

0102030405060708090

100

Family Voice/Choice Team Based Natural Supports Collaborative Community Based

Perc

ent F

idel

ity Care CoordinatorCaregiverTeam MemberYouth

Family Participation in Evaluation: How To Make It Work 163

Kathryn SchutteResearch and Training Center on FamilySupport and Children’s Mental Health(503) 725-8464kmschutt@pdx.edu

Paula SavageRegional Research Institute for Human Services

Adjoa RobinsonResearch and Training Center on FamilySupport and Children’s Mental Health

Pauline JivanjeeResearch and Training Center on FamilySupport and Children’s Mental Health

Michael PullmannRegional Research Institute for Human Services

Portland State UniversityPO Box 751Portland, OR 97207

Family Participation in Evaluation:How to Make it Work

In recent years, the use of participatory approachesin research and evaluation has gained momentum.Participatory approaches are believed to lead to studiesthat are more relevant to the needs of families.Ultimately, a goal of participatory approaches is toproduce findings that will be used to improve servicesfor families (Osher, van Kammen, & Zaro, 2001). TheNational Institute on Disability and RehabilitationResearch (NIDRR) has adopted a list of advantagesof participatory action research, formulated byTurnbull, Friesen, and Ramirez (1998):

1. Increased relevance of research to the concernsof family members;

2. Increased rigor of research;

3. Increased benefit to researchers in minimizinglogistical problems;

4. Increased utilization of research by families; and

5. Enhanced empowerment of researchers, familiesand other stakeholders (p.178).

Family involvement in children’s mental healthresearch and evaluation has received support from avariety of government agencies. NIDRR nowencourages grantees to use participatory actionresearch to involve families and consumers. Since1993, system of care projects funded by the Centerfor Mental Health Services’ ComprehensiveCommunity Mental Health Services for Children andTheir Families Program have been required to involvefamily members as participants in the evaluation ofthe projects. In addition, the Surgeon General’s Reporton Mental Health (U.S. Department of Health andHuman Services, 1999), the National Action Agendaon Children’s Mental Health (U.S. Department ofHealth and Human Services, 2001) and the President’s

164 Building on Family Strengths 2003 Conference Proceedings

National New Freedom Commission on MentalHealth (New Freedom Commission on Mental HealthSubcommittee on Consumer Issues, 2003) havepromoted family and consumer involvement in theresearch and evaluation of children’s mental healthservices.

These developments have presented family memberswith opportunities to work on teams that areevaluating the services their children are receiving.Evaluation teams at system of care grant communitiesaround the country employ family members on theirteams in different capacities. Involving familymembers on evaluation teams has led to thedevelopment of new family-evaluator partnerships insystems of care, requiring evaluators to collaboratewith family members in all aspects of the evaluationof services. Team members have reported a numberof benefits of involving family members on evaluationand research teams. For example, system of care teammembers have reported that family members are morelikely to gain the trust of research participants and toget accurate information from them (Osher &Telesford, 1996). In addition, family members haveassisted in the design of user-friendly formats forfindings, which facilitate the dissemination of resultsto wider audiences, especially other family members(Vander Stoep, Williams, Jones, Green, & Trupin,1999).

In addition to these benefits, there are also challengesto family participation in evaluation teams. While moreevaluators are now beginning to work with familymembers on evaluation teams, few have priorexperience working with these family members in thiscapacity. Many evaluators have been traditionallytrained to maintain control of the research processand distance themselves from the “subjects,” thosewho are being researched (Sohng, 1992). However, inthese family-evaluator teams, evaluators interact withfamily members as partners in the evaluation process.Without adequate training, the participatory approachcan present challenges to evaluators. There are alsochallenges related to the lack of organizationalsupports for family participation in evaluation teams.

This presentation discussed findings from a qualitativestudy of evaluators’ perspectives on collaboration withfamily members on evaluation teams. The purposeof the study was to gain understanding of thechallenges facing evaluators and family evaluatorsworking together and to identify effective strategiesfor building collaborative relationships in evaluationteams. Our presentation reported on evaluators’descriptions of challenges to collaboration and thestrategies they reported using to address them. Afamily member employed by Clark County,Washington’s system of care evaluation team describedexamples of the barriers the team has faced and thestrategies they used to address them.

METHOD AND SAMPLE

To find out evaluators’ perspectives about workingwith family members on evaluation teams, wedeveloped a qualitative survey. Open-ended questionswere designed to provide understanding of thechallenges facing evaluators and family membersworking together and effective strategies forcollaboration. We identified evaluators throughnominations by key informants and snowball samplingand conducted telephone or face-to-face interviewswith them.

We asked evaluators questions about the roles thatfamily evaluators have played on their evaluationteams, their training to work with family evaluators,the challenges they have faced, and the strategies theyhave used to overcome these challenges. Theinterviews were audio-taped, transcribed, and analyzedqualitatively using a grounded theory approach (Glaser& Strauss, 1967). Members of the team developed acoding scheme and independently coded the data,compared the coded sections of the interviews andnegotiated the categorization of the data. The datawas categorized according to themes, these themeswere synthesized, and examples were identified toillustrate the themes.

Results are based on interviews with 20 evaluatorsfrom different parts of the U.S., most of whom wereinvolved in the evaluations of systems of care fundedby the Center for Mental Health Services’

Family Participation in Evaluation: How To Make It Work 165

Comprehensive Community Mental Health Servicesfor Children and Their Families Program. Fourteenevaluators reported that they had Ph.D.’s, two werePh.D. candidates, and four held Masters degrees.Sixteen were female and four were male, with a meanage of 41.4 (SD = 11.2, range 28-66). These evaluatorsreported working in the field of children’s mentalhealth for an average of 13.8 years (SD = 7.4, range3-30) and 70 percent said they had some level ofexperience as a family member of a person withmental health needs or a consumer of mental healthservices.

RESULTS

Evaluators were asked how well their training hadprepared them for collaboration with family members.Fifty-five percent of the evaluators reported that weretrained to maintain distance from research participants.Most evaluators reported that partnering withstakeholders on research and evaluation teams wasnot discussed in their academic training. In responseto a question about how they learned to work withfamily members, 70 percent of the evaluators reportedthat they learned by doing the work. One evaluatorsaid that what helped her to do the work was, “learningby doing…learning from others around here who aretrying a lot of the same things.”

Family members were described as being involved ina variety of evaluation activities, including:

• helping with the development of projects,

• developing instruments/surveys,

• collecting data,

• training other family evaluators,

• participating in the decision making,

• assisting with analysis of data,

• assisting with the interpretation of results

• presenting data, and

• helping with dissemination of results.

In response to a question about challenges in theirwork with family evaluators, evaluators described

several difficulties. The primary challenges describedwere recruitment and hiring difficulties; payment ofadequate salaries; concerns about the impact of familymembers on research rigor; stakeholders’ lack ofappreciation for family evaluators’ roles; and thetension between advocacy and research. Respondentsalso discussed the strategies developed to address thesechallenges. Examples of the challenges and some ofthe strategies used to address them are summarizedbelow.

Difficulty recruiting and hiring family memberswithout relevant academic training

Some evaluators reported that it was difficult to hirefamily evaluators who did not have a degree due tohiring policies based on academic qualifications.Strategies that evaluators reported using were creatinga job classification specifically for family members andhiring family members as consultants. The ClarkCounty family evaluator reported that the team initiallyaddressed this problem by hiring a family member asa consultant through a contract with a family supportorganization. This organization did not have to adhereto the restrictive policies regarding hiring and pay thatconstrained the university-based research team.Subsequently, she was paid via a personal servicescontract with the university. Later, when she hadgained experience on the job, she was hired as auniversity-based research assistant.

Paying family members adequate salaries

Several evaluators mentioned their inability to payfamily evaluators salaries that were appropriate to thecomplexity of the work. As one evaluator commented,“We say on one hand that we value families as muchas the work of professionals, but we never pay themas much… it’s hard to pay them what they are worth.”To address this challenge, evaluators reporteddeveloping contracts to pay family evaluators basedon their skills and expertise, rather than based onacademic qualifications. One evaluator reported thatshe had worked to get the family evaluators a payincrease based on their skills and expertise rather thandegrees.

166 Building on Family Strengths 2003 Conference Proceedings

Concerns about the effects of family evaluators on the rigor ofthe research. Some evaluators reported stakeholders’concerns about the impact of family members withpersonal experience of receiving mental healthservices on the rigor of the research. They describedcommunity partners’ fears that family participationon the evaluation team would compromise the qualityof the research. A few respondents mentioned theirinitial fears that it would be difficult for familyevaluators to maintain objectivity. In response, anumber of evaluators commented that they hadlearned that obtaining objectivity is not really possible.One evaluator said, “I think that I was a little lessflexible about [objectivity] than I am now, realizingthat sometimes that fuzziness of boundaries actuallybenefits us researchers in a lot of ways.” The ClarkCounty family evaluator said that it had been necessaryto continually emphasize the benefits of having familymembers on the research team to communitystakeholders, while also encouraging objectivity.Evaluators reported that research rigor can bemaintained through frequent open discussions in teammeetings, training, sporadic supervision of interviews,and thorough verification of data. They suggested thatthese activities are especially important in the earlystages of research.

Community stakeholders’ lack of appreciationfor family evaluators’ roles

Several respondents reported that communitystakeholders, especially service providers, did not valuethe family member role on evaluation teams.Evaluators talked about the challenge of gainingrespect for the family members as real members ofthe evaluation team. In some cases, respondentsinformed us that family members gained respect overtime by demonstrating that they could do high qualitywork. Some evaluators reported educatingstakeholders about the valuable contributions offamily evaluators. One evaluator described helpingcommunity stakeholders to appreciate familymembers’ roles by demonstrating that familyparticipation improved the retention of participantsand quality of findings. The Clark County familyevaluator described the team’s efforts to gain respect

for her role, including “professionalizing” the familyevaluator role, mutual education, and naming everyoneon the team as team members, regardless of status.She reported that the team demonstrated theimportance of family members as core members ofthe evaluation team by ensuring that they attendmeetings, participate in presentations, and are involvedin interpretation of findings and writing reports.

Tension between advocacy andresearch rigor

Several evaluators mentioned challenges related tofamily evaluators’ desire to use preliminary data foradvocacy. One respondent reported that a familyevaluator became frustrated at seeing the data but notbeing able to use it. She said that she had talked withthis family member to help her to accept the slowerpace of change with research. Evaluators said thatopen communication about the tension betweenresearch and advocacy was helpful in dealing with thischallenge. For example, one evaluator stated, “Theother thing, I think, is the ability to negotiate things,like the advocacy and research issue. The willingness,we realize that we have to talk about this and we haveto hash it out.”

CONCLUSION

According to participants in this study, familymembers play a variety of roles on evaluation teamsat systems of care around the nation. Mostrespondents of this study reported that they had notreceived training on how to work with familymembers. Evaluators reported encountering a numberof challenges in their work with family evaluators andthey described strategies to overcome them. Next stepsin the research agenda are to expand ourunderstanding by exploring family evaluators’perspectives. We will examine evaluators’ trainingneeds related to collaboration and develop trainingmaterials to improve family-evaluator collaboration.

Family Participation in Evaluation: How To Make It Work 167

REFERENCES

Glaser, B. G., & Strauss, A. L. (1967). The discovery ofgrounded theory: Strategies for qualitative research.Chicago, IL: Aldine de Gruyter.

New Freedom Commission on Mental HealthSubcommittee on Consumer Issues. (2003). Reportof the Subcommittee on Consumer Issues: Shifting to arecovery-based continuum of community care. RetrievedMarch 13, 2003, 2003, from the World Wide Web:http://www.mentalhealthcommission.gov/subcommittee/Consumer_022803.doc

Osher, T., & Telesford, M. (1996). Involving familiesto improve research. In K. Hoagwood & P. S.Jensen & C. B. Fisher (Eds.), Ethical Issues in MentalHealth Research with Children and Adolescents (pp. 29-39). Mahwah, New Jersey: Lawrence ErlbaumAssociates.

Osher, T., van Kammen, W., & Zaro, S. M. (2001).Family participation in evaluating systems of care:Family, research, and service system perspectives.Journal of Emotional and Behavioral Disorders, 9, 63-70.

Sohng, S. S. L. (1992). Consumers as research partners.Journal of Progressive Human Services, 3, 1-14.

Turnbull, A. P., Friesen, B., & Ramirez, C. (1998).Participatory action research as a model forconducting family research. The Association forPersons with Severe Handicaps, 23, 178-188.

U.S. Department of Health and Human Services.(1999). Mental health: A report of the Surgeon General:Chapter 3: Children and Mental Health. WashingtonD.C.: U.S. Department of Health and HumanServices, Substance Abuse and Mental HealthServices Administration, Center for Mental HealthServices, National Institutes of Health, NationalInstitute of Mental Health.

U.S. Department of Health and Human Services.(2001). Report of the Surgeon General’s conference onchildren’s mental health: A national action agenda.Washington D.C.: U. S. Department of Health andHuman Services.

Vander Stoep, A., Williams, M., Jones, R., Green, L.,& Trupin, E. (1999). Families as full researchpartners: What’s in it for us? Journal of BehavioralHealth Services and Research, 26, 329-344.

Themes and Sample Items from the Community Supports for Wraparound Inventory

Item Fully developed system support Least developed system support Theme 1: Community Partnership. Collective community ownership of and responsibility for wraparound is built through collaborations among key stakeholder groups. (7 items)

Item 1.3 Influential Family Voice

Families are influential members of the community team and other decision-making entities, and they take active roles in wraparound program planning, implementation oversight, and evaluation. Families are provided with support and training so that they can participate fully and comfortably in these roles.

Family members are not actively involved in decision-making, or are uninfluential or "token" components of the community team, boards, and other collaborative bodies that plan programs and guide implementation and evaluation.

Theme 2: Collaborative Action. Stakeholders involved in the wraparound effort take concrete steps to translate the wraparound philosophy into concrete policies, practices and achievements. (8 items)

Item 2.3 Proactive Planning

The wraparound effort is guided by a plan for joint action that describes the goals of the wraparound effort, the strategies that will be used to achieve the goals, and the roles of specific stakeholders in carrying out the strategies.

There is no plan for joint action that describes goals of the wraparound effort, strategies for achieving the goals, or roles of specific stakeholders.

Theme 3: Fiscal Policies and Sustainability. The community has developed fiscal strategies to meet the needs of children participating in wraparound and methods to collect & use data on expenditures for wraparound-eligible children. (6 items)

Item 3.3 Collective Fiscal Responsibility

Key decision-makers and relevant agencies assume collective fiscal responsibility for children and families participating in wraparound and do not attempt to shift costs to each other or to entities outside of the wraparound effort.

Each agency has its own cost controls and agencies do not collaborate to reduce cost shifting, either to each other or to entities outside of the wraparound effort.

Access to Needed Supports & Services. The community has developed mechanisms for ensuring access to the wraparound process and the services and supports that teams need to fully implement their plans. (6 items)

Item 4.6 Crisis Response

Necessary support for managing crises and fully implementing teams' safety/crisis plans is available around the clock. The community’s crisis response is integrated with and supportive of wraparound crisis and safety plans.

Support for managing crises is insufficient, inconsistently available, or uncoordinated with wraparound teams' crisis and safety plans.

Theme 5: Human Resource Development & Support. The community supports wraparound and partner agency staff to work in a manner that allows full implementation of the wraparound model. (6 items)

Item 5.5 Supervision

People with primary roles for carrying out wraparound (e.g., wraparound facilitators, parent partners) receive regular individual and group supervision, and periodic "in-vivo" (observation) supervision from supervisors who are knowledgeable about wraparound and proficient in the skills needed to carry out the wraparound process..

People with primary roles for carrying out wraparound receive little or no regular individual, group, or observational supervision AND/OR supervisors are inexperienced with wraparound or unable to effectively teach needed skills.

Theme 6: Accountability. The community has implemented mechanisms to monitor wraparound fidelity, service quality, and outcomes, and to assess the quality and development of the overall wraparound effort.

Item 6.1 Outcomes Monitoring

There is centralized monitoring of relevant outcomes for children, youth, and families in wraparound. This information is used as the basis for funding, policy discussions and strategic planning

There is no tracking of relevant outcomes for children and youth in wraparound, or different agencies and systems involved maintain separate tracking systems.

RESOURCES – Family Partners and Data Driven Strategies

1. Articles and Manuals Slaton, E., (2004). Family Engagement in Evaluation: Lessons Learned, FFCMH http://www.ffcmh.org/evaluation.htm Crossbear, S., (2002) Families and Communities in Evaluation. Focal Point, Vol.16, No.2; pp.16-18 www.rtc.pdx.edu Jivanjee, P. & Robinson, A., (2007) Studying Family Participation in Systems of Care Evaluations; Journal of Behavioral Health Services and Research, pp.369-381 http://www.springerlink.com/content/831527m566l64852/ Bates, B., (2005). The Family-Driven Research Study: Lessons Learned by Families and Researchers Working Together. National Evaluation Team, ORC Macro United Advocates for Children and Families (2004). Evaluation Program Manual http://www.springerlink.com/content/831527m566l64852/

2. Discussion Board: National FFCMH Evaluation Discussion Room – FFCMH.org

3. Family organization tools and web based resources Western New York Nancy Craig, Parent Advisor NYS Office of Mental Health Western NY Field Office 737 Delaware Ave, Suite 200 Buffalo, NY 14209 716-885-4219 ext. 236 nncrg@adelphia.net New Jersey Nancy Fleming, Manager of ETO Implementation New Jersey Alliance of Family Support Organizations Family Support Organization of Essex County 60 Evergreen Place, suite 410 East Orange, NJ 07018 973-395-1441 flemingn@FSOEC.org Targeted Parent Assistance Jane Adams, Executive Director Keys for Networking, Inc. 211 West 33rd Street Topeka, KS 66611 Phone: 785-233-8732 jadams@keys.org

Maryland Jane Walker, Executive Director Md. Coalititon of FFCMH 10632 Little Patuxent Pkwy Suite 119, Columbia, MD 21044 410.730.8267 jwalker@mdcoalition.org Arkansas Pamela Marshall, Ar.FFCMH P.O. Box 56667 Little Rock, AR 72215 (501) 374-7218 pammarshall7218@sbcglobal.net Friesan, B., (2001) Family Participation Measure Regional Research Institute for Human Services, Research and Training Center on Family Support and Children’s Mental Health, Portland State University Portland, Oregon 97201 www.rtc.pdx.edu

Wraparound Fidelity Assessment System

The Wraparound Evaluation & Research Team c/o Eric J. Bruns, Ph.D., and April Sather, MPH

University of Washington School of Medicine 2815 Eastlake Ave. E., Suite 200, Seattle, WA 98102 | 206.685.2310

wrapeval@u.washington.edu

Description

Overview. The Wraparound Fidelity Assessment System (WFAS) is a multi-method approach to assessing the quality of individualized care planning and coordination for children and youth with complex needs and their families. WFAS instruments include interviews with multiple stakeholders, a team observation measure, a document review measure, and an instrument to assess the level of community and system support for wraparound. The instruments that comprise the WFAS can be used individually or, to provide a more comprehensive assessment, in combination with one another.

Uses. Fidelity measurement is a core implementation support to evidence-based practices. The WFAS provides a method for conducting fidelity measurement for the wraparound process, as specified by the National Wraparound Initiative. As a fidelity measurement system, WFAS instruments were designed to support both program improvement as well as research.

The WFAS tools were specifically designed to assess adherence to the 10 Principles of Wraparound and the Phases and Activities of the Wraparound Process as defined by the National Wraparound Initiative (www.rtc.pdx.edu/nwi).

With respect to program improvement, sites or programs delivering services via the wraparound process can generate profiles organized by the prescribed activities of the wraparound process or the 10 principles of wraparound, to illuminate areas of relative strength and weakness. This information can be used to guide program planning, training, and quality assurance.

With respect to research, data from WFAS instruments can help evaluate whether the wraparound process has been adequately implemented, and thus aid interpretation of outcomes. In addition, researchers on youth and family services may wish to use WFAS instruments to measure the relationship between adherence to the wraparound model and outcomes, as a way to explore which aspects of service delivery are most important to child and family well-being.

Other uses. Although the WFAS instruments were not intended originally for use on the individual family level, some programs use the measures to inform services for families, and supervision and coaching of individual staff and supervisors. However, great care is needed to insure confidentiality of the family and staff persons involved. Finally, WFAS instruments have not been used widely for standards conformance or certification assessment, but there has been some interest in adapting the WFAS measures for this purpose.

WFAS Instruments

2

Wraparound Fidelity Index, version 4.0 The measures that comprise the WFAS include: • The Wraparound

Fidelity Index, v.4 • The Team

Observation Measure • The Documentation of

Wraparound Process • The Community

Supports for Wraparound Inventory

The Wraparound Fidelity Index 4.0 (WFI-4) is a set of four interviews that measures the nature of the wraparound process that an individual family receives. The WFI-4 is completed through brief, confidential telephone or face-to-face interviews with four types of respondents: caregivers, youth (11 years of age or older), wraparound facilitators, and team members. It is important to gain the unique perspectives of all these informants to understand fully how wraparound is being implemented. A demographic form is also part of the WFI-4 battery.

The WFI-4 interviews are organized by the four phases of the wraparound process (Engagement and Team Preparation, Initial Planning, Implementation, and Transition). In addition, the 40 items of the WFI interview are keyed to the 10 principles of the wraparound process, with four items dedicated to each principle. In this way, the WFI-4 interviews are intended to assess both conformance to the wraparound practice model as well as adherence to the principles of wraparound in service delivery.

Team Observation Measure

The Team Observation Measure (TOM) is employed by external evaluators to assess adherence to standards of high-quality wraparound during team meeting sessions. It consists of 20 items, with two items dedicated to each of the 10 principles of wraparound. Each item consists of 3-5 indicators of high-quality wraparound practice as expressed during a child and family team meeting. Working alone or in pairs, trained raters indicate whether or not each indicator was in evidence during the wraparound team meeting session. These ratings are translated into a score for each item as well as a total fidelity score for the session overall.

Document Review Measure

The Documentation Review Measure (DRM) is a 32-item instrument that is used to assess the primary documentation requirements of high fidelity wraparound. The DRM is used by a trained evaluator who uses the tool to rate conformance to the principles of wraparound in materials such as the child and family’s wraparound plan, crisis and safety plans, transition plan, and meeting notes. Like the other WFAS fidelity tools, items on the DRM link to the 10 principles of the wraparound process, and result in scores for individual items, the 10 principles of wraparound, and a total score for the instrument overall. The DRM has been pilot tested and is currently under revision. A final version will be available as part of the WFAS in late 2008.

Community Supports for Wraparound Inventory

The CSWI is a research and quality improvement tool intended to measure how well a local system supports the implementation of the wraparound process. The CSWI is based on the framework of Necessary Conditions described by Walker, Koroloff and Schutte (2003), and presents 40 community or system variables that ideally are in place in communities that aim to implement the wraparound process. The CSWI is somewhat unique from the other WFAS instruments in that it assesses the system context for wraparound as opposed to the fidelity to the practice model for an individual child and family.

The CSWI can be used in several ways. First, it results in a quantified assessment of community supports for wraparound across multiple domains, so that researchers can determine the impact of these conditions on fidelity and outcomes. Second, it presents levels of support for multiple domains (e.g., funding, collaboration, accountability) so that evaluators and stakeholders can understand the full context for wraparound implementation. Third, results are structured

so that local groups can assess system supports for wraparound, respond to areas of strength and weakness, and monitor improvements over time.

CSWI domains include: • Community

Partnership • Collaborative

Action • Fiscal Policies

& Sustainability • Access to

Supports & Services

• Human Resource Development & Support

• Accountability

Psychometrics

Various versions of the WFI have demonstrated good test-retest reliability, internal consistency, and inter-rater reliability. Validity studies have found that fidelity as assessed by the WFI correlates with the ratings of an external wraparound expert, while other studies have found significant associations with child and family outcomes as well as the level of community supports for wraparound. The WFI-4 has been found to discriminate between wraparound and non-wraparound conditions as well as sites with different levels of implementation support for wraparound.

The fourth version of the WFI was created in order to bring the WFI in line with the specified practice model of the NWI. This revision to the WFI-4 also aimed to better operationalize its items to make it more reliable and valid. The TOM, DRM, and CSWI are newer measures that are being subjected to pilot testing and examination of reliability and validity. The Wraparound Evaluation and Research Team is currently seeking communities and programs interested in using these tools and participating in pilot testing.

Supporting Technologies

3

User’s Manuals are available for all instruments with detailed instructions and scoring rules. The WFI-4 and TOM also include a training toolkit for lead evaluators to use to train interviewers to criteria. Our Research Team has also developed data entry shells in SPSS and Excel formats for all WFAS measures and is completing work on a WFAS Online Data Entry and Reporting System (WONDERS) which will be available for use by all collaborating communities.

The WFI-4 has been translated into Spanish for use by collaborating communities who serve Spanish-speaking youth and families. If your program or community is interested in using one or more

of the WFAS instruments, please contact the Wraparound Evaluation and Research Team at wrapeval@u.washington.edu.

4

WFAS Instruments: Sample Items

Wraparound Fidelity Index, version 4

PHASE EXAMPLE QUESTION SCALE

PHASE I - Engagement

At the beginning of the wraparound process, did you have a chance to tell your wraparound facilitator what things have worked in the past for your child and family?

Yes/Somewhat/No

PHASE 2 - Planning

Are the supports and services in your wraparound plan connected to the strengths and abilities of your child and family?

Yes/Somewhat/No

PHASE 3 - Implementation Is there a friend or advocate of your child or family who actively participates on the wraparound team? Yes/Somewhat/No

PHASE 4 - Transition Has your team discussed a plan for how the wraparound process will end? (i.e., a “transition plan”) Yes/Somewhat/No

Team Observation Measure (TOM)

ITEM INDICATORS SCALE

Team Membership & Attendance Principle: Team based

a. Parent/caregiver is a team member and present at the meeting. b. Youth (over age 9) is a team member and present at the meeting. c. Natural supports for the family are team members and present. d. Key school or other public stakeholder agency representatives are present. e. Key service providers are team members and are present.

Y N N/A Y N N/A

Y N Y N N/A Y N N/A

Facilitator Preparation Principle: Collaborative

a. There is a clear agenda or outline for the meeting, which provides an understanding of the overall purpose of the meeting and the major sections of the meeting. b. The meeting follows an agenda or outline such that team members know the purpose of their activities at a given time. c. The facilitator has prepared needed documents and materials prior to the meeting. d. A plan for the next meeting is presented, including time & date.

Y N

Y N

Y N N/A

Y N N/A

Youth and Family Choice Principle: Voice and Choice

a. The youth prioritizes life domains, goals, or needs on which he or she would like the team to work. b. The caregiver or parent prioritizes life domains goals, or needs on which he or she would like the team to work. c. The family and youth have highest priority in decision making.

Y N N/A

Y N N/A

Y N N/A

5

Community Supports for Wraparound Inventory

ITEM FULLY DEVELOPED SYSTEM

SUPPORT 4 3 2 1 0 LEAST DEVELOPED SYSTEM SUPPORT

Community Team

There is a formal collaborative structure (e.g., a “community team” or other body) for joint planning and decision-making through which community partners take collective responsibility for development and implementation of wraparound.

4 3 2 1 0

The wraparound effort is unsupported by any collaborative system-level decision-making entity, leaving facilitators and other wraparound staff without a higher-level body to bust barriers or solve system-level problems on their behalf.

Full Agency Support

Relevant public agencies (e.g., mental health, child welfare, schools, courts) and major provider organizations all collaborate with and participate actively and productively on the community team. These agencies and organizations fully “buy in” to the wraparound effort.

4 3 2 1 0

Relevant child serving agencies and major provider organizations do not participate actively and constructively on the community team. The organizations or agencies that provide wraparound do so in the absence of "buy-in" from these other agencies and their staff.

Information Sharing

Information is shared efficiently across systems (or is maintained centrally for the wraparound program) so as to provide the data needed to monitor wraparound quality, plan implementation, costs, and outcomes.

4 3 2 1 0

Agencies have not resolved legal issues and/or developed mechanisms for efficiently sharing the information that is required to monitor wraparound quality, plan implementation, costs, and outcomes.

Fiscal Flexibility

Funds are available to pay for services and supports, and funds are flexible, so that teams can fully implement the strategies included in individual wraparound plans and safety/crisis plans.

4 3 2 1 0

Financing policies are rigid and are largely or entirely designed to facilitate payment for categorical services. There is little latitude for flexibility to provide creative, individualized care for children, youth, and families.

Document Review Measure SECTION ITEM SCORING

Strengths Discovery

There is evidence that the youth and family were engaged in the service delivery process within seven days of the referral.

3 – Met 2 – Mostly Met 1 – Some Met 0 – Not Met

Plan of Care

Planning and implementation with this child and family is being done through a child and family team.

3 – Met 2 – Mostly Met 1 – Some Met 0 – Not Met

Crisis Plan

The crisis plan includes strategies for preventing potential crises and is based on a comprehensive functional assessment.

3 – Met 2 – Mostly Met 1 – Some Met 0 – Not Met

Progress

There is evidence that the family and youth’s successes are identified and celebrated by the team.

3 – Met 2 – Mostly Met 1 – Some Met 0 – Not Met

Using Data to Implement Wraparound and Sustain Fidelity: Small Group Activity

1. Introduce yourselves to the members of your small group and pass around the sign-in sheet.

2. Appoint a recorder and reporter.

The recorder should record all of the responses to the next two items and turn in to one of the presenters. We will type this up and post on the web, so make your record as complete as possible.

3. Discuss the experiences your communities have had in using data to assess

wraparound quality and/or fidelity, or using skill sets and credentialing to guide wraparound implementation.

• List things that seem to be working and how it helps.

• List challenges that have been encountered.

• As a group, propose solutions to these challenges. (You may not have

solutions for every challenge—that’s ok.)

4. Discuss how the tools you have heard about could help you improve practice or

implementation quality in your community AND/OR why these tools wouldn’t meet your community’s needs (and what type of tool or approach might work, or how the existing tools might need to be adapted).

5. Report out. Select up to

a. Two examples of what’s working b. One example of a challenge--with or without a solution--that your group

has thought of c. Up to two ideas (total) about

i. How the tools could help OR ii. Why these tools might not work (and what type of tool or approach

might work, or how the existing tools might need to be adapted).

6. Turn in your group record.