if training is the answer, what’s the question? toward an evidence-based system for innovation...

IF TRAINING IS THE ANSWER, WHAT’S THE QUESTION?

TOWARD AN EVIDENCE-BASED SYSTEM FOR INNOVATION

SUPPORTAbraham Wandersman

U. Of South Carolinawandersman@sc.edu

QUESTIONS

• IF TRAINING IS THE ANSWER, WHAT’S THE QUESTION?

• IF TRAINING EVALUATION IS THE ANSWER, WHAT’S THE QUESTION?

OVERVIEW

• AMBITIOUS GOALS• THE INTERACTIVE SYSTEMS FRAMEWORK FOR

DISSEMINATION AND IMPLEMENTATION• DEMYSTIFYING ACCOUNTABILITY AND

GETTING TO OUTCOMES ® (TAKING A VACATION)

• TOWARD AN EVIDENCE BASED SYSTEM FOR INNOVATION SUPPORT

• SUMMARY (JEOPARDY)

AMBITOUS GOALS

• Dr. von Eschenbach: I believe we are at what I call a strategic inflection in biology, which means we're at a point of unprecedented growth in three key areas related to cancer research: knowledge, technology, and resources. The integration of growth in these three sectors provides an opportunity for exponential progress. To achieve this progress, we must set a clear direction and focus our efforts into a cohesive strategy.

• The goal of eliminating suffering and death due to cancer provides this focus. It does not mean "curing" cancer but, rather, it means that we will eliminate many cancers and control the others, so that people can live with -- not die from -- cancer. We can do this by 2015, but we must reach for it. We owe it to cancer patients around the world -- and their families -- to meet this challenge.

May 16, 2003 BenchMarks

ADDITIONAL AMBITIOUS GOALS

• SAFETY• PERMANENCY• CHILD WELL-BEING

DATA - EVIDENCEDATA - EVIDENCE

THE GAP BETWEEN RESEARCH AND PRACTICE

• IN THE DOCTOR’S OFFICE

• OVERALL 54.9% RECEIVED RECOMMENDED CARE

ASCH ET AL STUDY, NEJM, 2006

POSSIBLE SOLUTION

• VA MEDICAL SYSTEM HAS 67% RECOMMENDED CARE

SYSTEM HAS ELECTRONIC MEDICAL RECORDS, DECISION SUPPORT TOOLS, AUTOMATED ORDER ENTRY, ROUTINE MEASUREMENT AND REPORTING ON QUALITY, INCENTIVES FOR PERFORMANCE

As Yogi Berra supposedly said,

"In theory there is no difference between theory and practice, but in practice there is."

Gates Foundation

Preventive Intervention

Vaccine/Drug

Mechanism

Syringes

Physician

Health System

Support System

Medical Schools

Government Funding

Traditional Evaluation Model

Agency

Funder

Evaluator

Funder

Partnership Model

RESULTS

Funder

Evaluator

Agency

HOW DO WE GET FROM HERE TO THERE ?

THE INTERACTIVE SYSTEMS FRAMEWORK FOR

DISSEMINATION AND IMPLEMENTATION

io

Distilling the Information—Prevention Synthesis & Translation System

Supporting the Work—Prevention Support System

Putting It Into Practice—Prevention Delivery System

Synthesis

General Capacity Building

Innovation-Specific Capacity Building

General Capacity Use

Innovation-Specific Capacity Use

Macro Policy

Climate

Funding

Existing Research and Theory

Translation

Support System Model

=+To

AchieveDesired

Outcomes

Actual Outcomes Achieved

Current Level of Capacity

Innovation

Training

TA

QI/QA

Tool

TOOLS

TRAINING

TECHNICAL ASSISTANCE

QUALITY IMPROVEMENT/QUALITY ASSURANCE

DEMYSTIFYING ACCOUNTABILITY

GTO®

Accountability Questions Relevant Literatures

1. What are the underlying needs and conditions that must be addressed? (NEEDS/RESOURCES)

1. Needs/Resource Assessment

2. What are the goals, target population, and objectives? (i.e., desired outcomes)? (GOALS)

2. Goal Setting

3. What science (evidence) based models and best practice programs can be used in reaching the goals (BEST PRACTICE)?

3. Consult Literature on Science Based and Best Practice Programs

4. What actions need to be taken so the selected program “fits” the community context? (FIT)

4. Feedback on Comprehensiveness and Fit of Program

5. What organizational capacities are needed to implement the program? (CAPACITIES)

5. Assessment of Organizational Capacities

6. What is the plan for this program (PLAN) 6. Planning

7. Is the program being implemented with quality (PROCESS)

7. Process evaluation

8. How well is the program working? (OUTCOME EVALUATION)

8. Outcome and Impact Evaluation

9. How will continuous quality improvement strategies be included? (IMPROVE)

9. Total Quality Management; Continuous Quality Improvement

10. If the program is successful, how will it be sustained? (SUSTAIN)

10. Sustainability and Institutionalization

GTO as a Painter’s Palette

#1Needs/

Resources

#2Goals

#3Best

Practices

#9Improve/

CQI#8

OutcomeEvaluation

#7Implementation

Process Evaluation

#4Fit

#5Capacities

#6Plan

#10Sustain

RESULTS

VACATION EXAMPLE

What Is Getting To Outcomes-2004?

• By Matthew Chinman, Pamela Imm & Abraham Wandersman

• A system based on ten empowerment evaluation and accountability questions that contain elements of successful programming

• Published by the RAND Corporation (quality review)

• Available at no cost at: http://www.rand.org/publications/TR/TR101/

• “Best Practice Process” –WINNER OF 2008 OUTSTANDING PUBLICATION AWARD

• DOWNLOADED OVER 75,000 TIMES

• By Matthew Chinman, Pamela Imm & Abraham Wandersman

• A system based on ten empowerment evaluation and accountability questions that contain elements of successful programming

• Published by the RAND Corporation (quality review)

• Available at no cost at: http://www.rand.org/publications/TR/TR101/

• “Best Practice Process” –WINNER OF 2008 OUTSTANDING PUBLICATION AWARD

• DOWNLOADED OVER 75,000 TIMES

Levels & AccountabilityCountry State County Agency Provider

Accountability Question

1. NEEDS/ RESOURCES

2. GOALS

3. EVIDENCE-BASED PRACTICES

4. FIT

5. CAPACITY

6. PLAN

7. IMPLEMENTATION

8. OUTCOME EVALUATION

9. CQI

10. SUSTAINABILITY

TOWARD AN EVIDENCE BASED SYSTEM

FOR INNOVATION SUPPORT (EBSIS)

WANDERSMAN, CHIEN, KATZ(ACCEPTED WITH REVISIONS)

EVIDENCE BASED SYSTEM for INNOVATION SUPPORT

=+To

AchieveDesired

Outcomes

Actual Outcomes Achieved

Current Level of Capacity

Innovation

Training

TA

QI/QA

Tool

AN ILLUSTRATION OF EBSIS

• THE MOMS PROJECT FOR SUBSTANCE ABUSING PREGNANT WOMEN

Core Components of MOMS program

• Client-Centered Care• Getting to Outcomes ®• Motivational Interviewing• Rewards system

EBSIS Components

• Tools• Training• Technical Assistance• Quality Assurance/Quality Assistance

Tool: MOMS Manual• Developing Fall 2009-Spring 2011• Three versions: Clinician, supervisor, and client• Described:– GTO process for individual treatment planning and

deliver– Use of assessment tools for planning, monitoring, and

evaluating client progress– Use of motivational interviewing to move clients

through the GTO Steps• However, we know that many tools end up sitting

on shelves!

Training for Outcome (TFO)

• Developing in conjunction with LRADAC leadership and clinicians

• Utilized thorough assessment of training needs to assure training transfer

• Separated into modules tailored to specific job descriptions

• Delivered 10.21.10-10.22.10 to over 70 staff!

Technical Assistance

• Began immediately following training• Implementation planning January 2011– Using the Quality Implementation Tool (QIT)

• Regular meeting with clinical/supervisory staff around imp. Issues and program refinement

• Clinician booster session re: use of MI and other MOMS components 2.11

QI/QA

• Monitoring process and Evaluating Outcomes• Assessment of Information System Capacity• Streamline collection and utilization of

information• Designed to:– Inform Continuous Quality Improvement (CQI)– Evaluate multi-dimensional outcomes

ENHANCING THE SCIENCE AND PRACTICE OF INNOVATION SUPPORT

• ZOOM OUT--THE 40 CELLS OF EBSIS

Tools Training TA QI/QANeeds & Resources

Goals

Best Practices  

Fit  

Capacity  

Plan

Implement/Process Evaluation

Outcomes

CQI

Sustainability

ZOOM IN--GTO AND WEBINARS

GTO Step  1. Need and Resources Identification of group to be targeted by the webinar

What does this group need to know?o Conduct a task analysis to determine the focus of webinar

Conduct a value analysis to determine whether presenting issues are worth addressing o Use cost-benefit analysis to determine if perceived benefits of training outweigh

anticipated training costs in terms of financial and resource investment.

1. Goals and Objectives Determine training goals based on needs/resource assessment datao What knowledge or skills will participants gain? (Webinars are more useful for

imparting conceptual or basic procedural knowledge – as opposed to hands-on/skill building-oriented knowledge)

Ensure training goals are aligned with organization’s goals and culture

1. Best Practices Webinar Design and DeliveryEvent management capabilities (e.g., GoToMeeting, Elluminate, AdobeConnectNow) can increase efficiency by automating:

o invitations to the webinar, registration processes, meeting reminders, real-time assessments of satisfaction during webinar, follow-ups and post-webinar feedback surveys, annotations and highlighting procedures, and sharing of documents

Commencement of webinar one or two minutes subsequent to the scheduled starting time in order to maximize attendanceReduction of background noise is important; participants should be instructed to keep their phones on mute until they have a question or a commentPresentation of an outline of the webinar as an introductory PowerPoint slideCounterbalancing of graphics with words in Webinar slides; graphics are important as 55% of learners are visual learners Limit the participant group to 15 to 17 people, especially when training on a new skill. This allows for high levels of interaction and enables the instructor to identify people who may be struggling with the content.Webinar ContentContent of Webinar should align with best practice literature in content area

1. Fit Customization of webinar advertisements to the population of focus Customization of webinar content to the population of focus

1. Capacities Technical capacitiesDoes the audience have access to computers, internet, space free from distractions, etc.?How many simultaneous users does Webinar software support?Human capacitiesIdentification of individuals for webinar roles: primary presenter, marketer, moderator, and technical coordinator

1. Plan Scheduling of date and time for webinar Distribution of personalized invitations to potential attendees Development of a schedule that outlines the sequence of a webinar’s components or phases Develop webinar content and other materials Rehearse webinar presentation Distribution of reminder emails to individuals who are registered for webinar Maintenance of a library with recorded webinars; individuals who registered for the webinar but no-

showed should be notified about the availability of a recording1. Process Evaluation Monitoring of implementation metrics, including number of registrants, and attendance rate

Monitoring of attendees’ engagement o Use of real-time polls to get feedback about participants’ opinions and attitudeso Use of a real-time “satisfaction meter” as an index of audience engagement

Administration of post-webinar surveyo Collection of Likert self-report data from attendees on: webinar content, webinar design, presenter

effectiveness (clarity and pace), planning and delivery Facilitator debriefings

1. Outcome Evaluation Were Goals and Objectives from Step 2 met?

1. CQI Develop a systematic process for learning from previous webinars to improve future webinars.

1. Sustainability Identify program champions Address funding and marketing issues

Distilling the Information—Prevention Synthesis & Translation System

Supporting the Work—Prevention Support System

Putting It Into Practice—Prevention Delivery System

Synthesis

General Capacity Building

Innovation-Specific Capacity Use

General Capacity Use

Innovation-Specific Capacity Use

Climate

Funding

Existing Research and Theory

Translation

Implem

entation

Implementation

QI/QA

Macro Policy

iGTO

GTO Contracting

Outcomes

• WHAT CAN FUNDERS, PRACTITONERS, AND RESEARCHERS/EVALUATORS HAVE IN COMMON?

Accountability Partnership Model

RESULTS

Funder

Evaluator

Agency

As Yogi Berra supposedly said:

As Yogi Berra supposedly said:

“It’s déjà vu all over again.”

SUMMARY

JEOPARDY

AN ANSWER 1) A COST-EFFICIENT COMPONENT FOR

INCREASING CAPACITY FOR KNOWLDEGE, SKILLS, AND BEHAVIOR WITHIN A BROADER SUPPORT SYSTEM

2) ACCOUNTABILITY PROGRAM IMPROVEMENT

WHAT IS THE PURPOSE OF TRAINING?

• A COST-EFFICIENT COMPONENT FOR INCREASING CAPACITY FOR KNOWLDEGE, SKILLS, AND BEHAVIOR WITHIN A BROADER SUPPORT SYSTEM

WHAT IS THE PURPOSE OFTRAINING EVALUATION?

• ACCOUNTABILITY • PROGRAM IMPROVEMENT

STIMULUS QUESTIONS

• What is the purpose of IV E training? (eg what do you hope will be accomplished by IV-E training)

• What is the purpose of training related to CFSR?

As Yogi Berra supposedly said,

"If you see a fork in the road, take it."

REFERENCES• Chinman M, Imm P, Wandersman A (2004). Getting to Outcomes 2004:

Promoting Accountability Through Methods and Tools for Planning, Implementation, and Evaluation. Santa Monica, CA: RAND Corporation, TR-TR101. Available at http://www.rand.org/publications/TR/TR101/.

• Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008). Bridging the gap between prevention research and practice: The Interactive Systems Framework for Dissemination and Implementation. American Journal of Community Psychology, 41, 171-181.

• Wandersman, A., Chien, V., & Katz. J. (accepted with revisions) Toward an Evidence Based System for Innovation Support (Tools, Training, Technical Assistance and Quality Improvement/Quality Assurance. American Journal of Community Psychology.