national child traumatic stress initiative (nctsi) data and evaluation webinar
TRANSCRIPT
National Child Traumatic Stress Initiative
(NCTSI)
Data and Evaluation Webinar
Webinar Overview
Overview of data and evaluation for the NCTSI: National Center for Child Traumatic Stress
(NCCTS) Data and Evaluation Program/Core Data Set (CDS)
Cross-site Evaluation Transformation Accountability (TRAC)
NCCTS Data and Evaluation Program
Range of quality improvement, data collection, and evaluation initiatives CDS:
Captures quantitative data on Network-supported treatments and services
Types of traumatic events Treatments delivered Functioning of youth over time
Cross-site Evaluation
Eight study components: Child and family outcomes and satisfaction with
services Network functioning with regard to trainings,
collaboration, product development, and product adoption and dissemination
TRAC
TRAC: Uniform tool for collecting data on mental health
outcomes across programs Grantees are required to collect TRAC data as
part of SAMHSA’s reporting obligations
NCCTS Data and Evaluation NCCTS Data and Evaluation Program: Overview and Next Program: Overview and Next
StepsStepsErnestine Briggs-King, Ph.D.National Center for Child Traumatic StressDuke University School of Medicine
Objective: How Do All the NCTSI Objective: How Do All the NCTSI Data Collection Initiatives Fit Data Collection Initiatives Fit
Together?Together?
SAMHSA initiatives
Data & Evaluation Program Initiatives
MACRO International
Initiatives
NCTSN Member sponsored initiatives
QPR
Core Data SeteSUF, Metrics,Product EvaluationCross-site Evaluation
TRACSurveys
Data and Evaluation ProgramData and Evaluation ProgramErnestine Briggs-King, Ph.D.919-682-1552, ext. 254
Bart Evans, B.S.Project [email protected]
Rebecca Wilgus, RN, [email protected]
Data and Evaluation Program: What Data and Evaluation Program: What Do We Do?Do We Do?
The Data and Evaluation Program is broadly concerned with data collection, analysis, and dissemination.
The Duke Clinical Research Institute (DCRI) provides the following services:●Data operations (includes project management, regulatory, contracts, finance)●Data management●Data coordinating activities
The NCTSN Data Repository is housed at the DCRI.
Data Operations Committee includes key members from NCTSN Centers, SAMHSA, Macro, UCLA, and Duke.
Regulatory IssuesData Collection Initiatives
●Electronic Service Utilization Form (eSUF) • Aggregate data on clients, services, and trainings
●NCTSN member-driven data collection initiatives• Surveys (e.g., New Grantee Surveys)• Product evaluation • Collaborative group initiatives
●Evaluation • Training, implementation, and dissemination efforts
●Core Data Set (CDS)• Client-level data about youth and families served
Data and Evaluation Program: What Data and Evaluation Program: What Do We Do?Do We Do?
Introduction: CDSIntroduction: CDS Quality improvement initiative Network-wide data collection that will provide
answers to central Network questions Essential for ensuring that the work done in
the Network is systematically measured, disseminated, and recognized
Numerous benefits for participating clinicians, individual sites, and the Network as a whole●Clinical improvements and utility●Real-time reports●Sustainability
NCTSN Clinical Summary Report
What is in the CDS?What is in the CDS? Demographic and living situation information Trauma history and detail Indicators of severity Treatment services and interventions Standardized assessment measures
Behavioral and Emotional Difficulties ●Child Behavior Checklist●UCLA PTSD Reaction Index●Trauma Symptom Checklist for Children-Alternate
CDS measures: administered at treatment entry, end of treatment (if short-term) or every 3 months
Bidirectional Data FlowBidirectional Data Flow●All centers will have
continuous (interactive) access to client-level reports (i.e., scoring of measures, summaries of client history)
●All centers will have continuous (interactive) access to data status reports that describe data entered by their center
●Each center can receive from Data Repository a ready-to-analyze data set of its own data
Next Steps: Data CollectionNext Steps: Data CollectionData collection and data entry resource requirements are determined by your implementation strategy
●Real-time—Clinicians collect and enter data online during assessment of client and family
●Ex post facto—Clinicians collect data on paper during their assessment, and data entry staff enter data into InForm at a later date
●Hybrid model—Combination of above models
What will work best at your center??
Next Steps: Regulatory Processes Next Steps: Regulatory Processes
Start the regulatory process soon!!• Assign a “point person” • Obtain documentation from your IRB• Submit a Public Health Information Letter • Signed Data Use Agreement • Templates, roadmaps and technical assistance are available
- Web site: www.nctsn.org - Data and Evaluation staff: Just a phone
call away! If your center is not affiliated with a local IRB, the DUHS IRB may serve as your IRB of record. Contact Bart Evans at the Data and Evaluation Program for more information.
Next Steps: TrainingNext Steps: Training
Training—Technology How to use the InForm system
Training—Clinical Content How to complete the case report
forms How to administer the
standardized assessments
NCTSN Data and Evaluation NCTSN Data and Evaluation Program’s Program’s
Commitment to YouCommitment to You• Customer support DCRI EDC Help Desk: [email protected]
Phone: 888-372-7743
NCTSN Data and Evaluation Team: [email protected]
Phone: 919-668-8182
• Technical consulting and training
• Clinical expertise and guidance
Data and Evaluation Webinar
Overview: Data Collection and Evaluation Activities
Introduction to Macro
Macro International • Management and consulting research
organization– More than 1,000 employees– Multiple offices in the United States and overseas– Many contracts with local, State, and Federal
governments (80–90 percent of Macro’s work)
• More than 40 years as a government contractor– Applied health services research– Information technology– Organizational development– Communications
20
21
The Common Goal
To raise the standard of care and improve access to services throughout the United States for children and adolescents and their families who have experienced trauma
Monitoring and Evaluating the NCTSN
Evaluation has always played a large role in the NCTSN
Initial congressional authorization required a “rigorous evaluation plan” for each center (P.L. 106-310, 2001)
Evaluation plans were to include methods for assessing the effectiveness of:• Processes• Treatment/intervention outcomes
22
What is Cross-site Evaluation (CSE)?
23
Network Collaboration
National Impact NREPP
Descriptive and Clinical
Outcome Study
Satisfaction Study
Trauma- Informed Services
Adoption and Implementation
CLIE
NT &
FA
MIL
Y
LEV
EL
NETW
OR
K L
EV
EL
Product Development
and Disseminatio
n Study
PROVIDER LEVEL
Macro wins competitive bid to conduct CSE
Macro begins year of evaluation design, development, and review
History of NCTSN Growth andCSE Participation
24
CSE implementation ongoing 44 grantees currently funded 70 grantees ever funded
OMB approves CSE design Macro begins implementing CSE 44 grantees currently funded 70 grantees ever funded
16 new grantees join NCTSN
12 grantees continue on consecutive awards
44 grantees currently funded
70 grantees ever funded
53 grantees currently funded
36 grantees currently funded
20 grantees currently funded
NCTSI program authorized under Children’s Health Act of 2000
10 new grantees join NCTSN 5 grantees continue on
consecutive awards 44 grantees currently funded 80 grantees ever funded
2000 2001 2002 2003 2004 2005 2006 2007 2008
Fiscal Years
Note: Counts of “currently funded” grantees exclude the National Center for Child Traumatic Stress
5 new centers join NCTSN 3 grantees continue on
consecutive awards 54 grantees currently
funded 85 grantees ever funded
Steps Toward Implementation
Ongoing Technical Assistance
25
Continuum of Training andTechnical Assistance
MONTHLY UPDATES
ReviewingDetails of theCross-site Evaluation
Creating Center-based Strategies
for Implementation
Regional Training
EVALUATION LIAISON SUPPORT
Component-specific
Updates by E-mail/Ongoing
Consultation
Cross-siteProcedures
Manual
Individual Training and
Technical Assistance
Getting Started: PracticalHelp Tips
Getting Started
27
Engaging Stakeholders in the Evaluation • Staff members• Community members• Other relevant stakeholdersEnsuring Organizational Readiness• Staff hiring plan and roles• Staff orientation to grant activities • Communication with current granteesBeginning Institutional Review Board (IRB)
Process• Local IRB plans• Local IRB application process and deadlines
28
Study Component Data Sources Who Collects
Data Who
Responds Timeframe for Implementing
Local IRB
1. Descriptive and Clinical Outcomes of Children Receiving Direct Clinical Mental Health Services
Core Clinical Characteristics Forms*
Center Family consumers FY 2006 and
ongoing thereafter Involves local IRB submission
TSCC-A*
UCLA PTSD Index for DSM-IV*
CBCL (1.5-5; 6-18)*
2. Consumer Satisfaction With Direct Clinical Mental Health Services
YSS-F Macro Family consumers FY 2006 and
ongoing thereafter
Depends somewhat on
local IRB; however, likely no implications
3. Provider Knowledge and Use of Trauma-informed Services
Key informant interviews and focus groups
Macro
Center-defined experts
FY 2006 and early FY 2007
TIS Survey Providers FY 2008 and
ongoing thereafter
4. Product/Innovation Development and Dissemination
Centers’ quarterly progress reports and annual progress reports
Macro
Administrators FY 2006 and
ongoing thereafter Collaborative group leader
interviews Collaborative group
leaders FY 2006 and FY 2008
Case studies Network staff, non-
Network staff FY 2007 and FY 2009
5. Adoption of Methods and Practices
GAAS Macro
Administrators, providers, evaluators
FY 2006 and annually thereafter
AIFI Administrators,
providers FY 2008 and
annually thereafter
6. Network Collaboration Network Survey
Macro Administrators FY 2006 and FY 2008
CTPT survey Collaborative group
members FY 2007 and FY 2009
7. National Impact of the NCTSI
National Impact Survey Macro Professional organization
members
FY 2006 and annually thereafter
No implications
8. NREPP Incorporated into regular Network
reporting mechanisms Macro Not applicable
Ongoing monitoring process
No implications
Getting Started (cont.)
Contact Information • If you have questions or comments or need more
information, feel free to contact the following members of the CSE team:–Christine Walrath, Officer in ChargeE-mail: [email protected]–John Gilford, Project DirectorE-mail: [email protected]–Bhuvana Sukumar, Deputy Project DirectorE-mail:[email protected]
29
Center for Mental Health Services (CMHS)
and Performance Management
NCTSINCTSI CMHS Transformation CMHS Transformation Accountability (TRAC)Accountability (TRAC)
AgendaAgenda
• Performance management and CMHS
• TRAC and NCTSI
• Next steps
CMHS Performance ManagementCMHS Performance Management
Federal/AnnualGPRA
PART Review
National Outcome Measures (NOMs)
TRAC
Eval
Federal/4 yrs
SAMHSA
CMHS
NCTSI
TRAC OverviewTRAC Overview
CMHS-TRAC is a Web-based, centralized, data platform that collects and reports outcome measures on CMHS programs for the Government Performance and Results Act (GPRA).
TRAC:TRAC: A New Approach to Accountability
• TRAC is a strategic imperative for CMHS
• Driven by:
– Government-wide requirements
– SAMHSA data strategy
– Center commitment to performance management
• Outcome measures targeted at:
– Consumer services
– Technical assistance
– Infrastructure development
– Prevention
SAMHSA-wide Domains:NOMs
SAMHSA is obtaining OMB approval to use NOMs for GPRA SAMHSA is obtaining OMB approval to use NOMs for GPRA reporting purposes:reporting purposes:
– Functioning – Stability in Housing– Employment and Education– Crime and Criminal Justice Status– Perception of Care– Social Connectedness – Access/Capacity– Retention– *Cost-effectiveness– *Use of Evidence-based Practices
*Indicates that measures have not yet been developed for these domains
CMHS GPRA Data Collection:CMHS GPRA Data Collection:Current StatusCurrent Status
Most programs were collecting GPRA data in some form
Lack of standardized outcome data:– Limits the ability to demonstrate what CMHS
has achieved– Creates difficulties in aggregating data to
summarize program performance NOMs tool is now the standard data
collection method for all service programs
TRAC and NCTSI TRAC and NCTSI Service Treatment GrantsService Treatment Grants
• NOMs tool used to collect interviews on all consumers enrolled in services: – Paper NOMs tool/electronic NOMs tool
• Data collection points—baseline, reassessment, and discharge
• Reassessment is collected every 3 months while consumer is receiving services
• No followup after discharge or if consumer is lost to contact for 90 days or more
What Can TRAC Do for You?What Can TRAC Do for You?
Reports:
– Enrollment and reassessment rates
– NOMs Outcome Report
– Cross Tabs and Frequencies Data download:
– Grantees can download data through CMHS-TRAC system and run analysis
How Can Grantees Prepare?How Can Grantees Prepare?
• Visit the TRAC Web site for more information at https://www.samhsa-gpra.samhsa.gov/home/ index.htm (Click CMHS TRAC General Information Service Programs Related Links )
• Contact TRAC Help Desk with your questions at 888-219-0238 or [email protected]
• If necessary, seek IRB approval immediately
• Determine processes for data collection
Q & AQ & A