dean l. fixsen, karen a. blase, michelle duda, sandra naoom, melissa van dyke, frances wallace...

Dean L. Fixsen, Karen A. Blase,

Michelle Duda, Sandra Naoom,

Melissa Van Dyke, Frances WallaceNational Implementation Research Network

Louis de la Parte Florida Mental Health Institute

Effectively Using Innovations in Education

MiBLSi Meeting 2008

Copyright © Dean L. Fixsen and Karen A. Blase, 2008

Evidence-Based Movement

The “evidence-based movement” is an international experiment to make better use of research findings in typical service settings.

The purpose is to produce greater benefits to consumers and society.


Education

65 million kids

6 million teachers and staff

100,000+ schools

3,143 counties

60 states & U.S. jurisdictions


Science to Service

SCIENCE SERVICEGAPIMPLEMENTATION


Science to ServiceScience to Service Gap

What is known is not what is adopted to help children, families, and caregivers

Implementation Gap What is adopted is not used with fidelity

and good outcomes for consumers.

What is used with fidelity is not sustained for a useful period of time.

What is sustained is not used on a scale sufficient to impact social problems.


Implementation

Since the 1960s there have been significant advancements in the science and practice of implementation

These advancements have many implications for making changes and improvements in human services


“Letting it happen”

Directives (e.g. laws, regulations), diffusion, and dissemination of information (e.g. publications, meetings, manuals)

Recipients are accountable for making use of the information to the extent they can

Based on Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004


“Helping it happen”

Dissemination and active promotion via meetings, conferences, training to provide more detailed information, and some skill building and consultation on the process

Recipients are accountable for making use of the information and training to the extent they can



“Making it happen”

Implementation: Active involvement of implementation teams that work at the intersection of practices, programs, systems, communities, & scientists

Implementation teams are accountable for assuring use of innovations with fidelity and good outcomes



0

100

200

300

400

500

600

700

800

900

≤1972 1974 1976 1978 1980 1982

0

50

100

150

200

250

300

CumulativeCouples

CumulativeHomes

Fixsen, Blase, Timbers, & Wolf (2001)

Teaching–Family Model


Follow Through Programs

Figure 1: This figure shows the average effects of nine Follow Through models on measures of basic skills (word knowledge, spelling, language, and math computation), cognitive-conceptual skills (reading comprehension, math concepts, and math problem solving) and self-concept. This figure is

adapted from Engelmann, S. and Carnine, D. (1982), Theory of Instruction: Principles and applications. New York: Irvington Press.


School Wide PBS

SWPBS Schools

0

1000

2000

3000

4000

5000

6000

7000

Aug 04 Aug 05 Aug 06 Aug 07

Year

Tota

l Num

ber

Sch

ools

4.2% of all schools


Implementation Reviews

Human service prevention and treatment programs (e.g. substance abuse, adult / children’s MH, justice, health, education)

Advanced manufacturing technologies

AMA clinical guidelines

Engineering: bridge maintenance

Hotel service management

National franchise operations

Cancer prevention & treatment


Ineffective MethodsExcellent experimental evidence for

what does not work

Diffusion/dissemination of information by itself does not lead to successful implementation (research literature, mailings, promulgation of practice guidelines)

Training alone, no matter how well done, does not lead to successful implementation


Ineffective MethodsExcellent evidence for what does

not work

Implementation by edict by itself does not work

Implementation by “following the money” by itself does not work

Implementation without changing supporting roles and functions does not work Paul Nutt (2002). Why Decisions Fail


What Works

Effective intervention practices

+

Effective implementation practices

=

Good outcomes for consumers


Implement Innovations

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION

INT

ER

VE

NT

ION Student Benefits


Implementation

An effective intervention is one thing

Effective implementation of an effective intervention is a very different thing


EBPs & Implementation

From an implementation perspective, what do we need to know about innovations such as evidence-based programs?



The usability of a program has little to do with the quality or weight of the evidence regarding that program

Evidence on intervention effectiveness for specific populations helps us choose what to implement

Evidence on the effectiveness of the intervention does not help implement the program or practice successfully



Core intervention components

Clearly described (who/what)

Practical measure of fidelity (critical)

Fully operationalized (teachable)

Field tested (practicable; complete)

Contextualized (org./systems fit)

Effective (worth the effort)


Implementation

What do we need to know about successful implementation methods?


Stages of Implementation

Implementation is not an event

A mission-oriented process involving multiple decisions, actions, and corrections

“Pursue to a conclusion” dictionary.com



Exploration

Installation

Initial Implementation

Full Implementation

Innovation

Sustainability

Implementation occurs in stages:

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

2 – 4 Years



Exploration

Installation

Initial Implementation

Full Implementation

Innovation

Sustainability

Implementation occurs in stages:

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

2 – 4 Years

Intervention Outcomes

0% 100%

Implementation Outcomes


INTEGRATED & COMPENSATORY

CONSULTATION & COACHING


STAFF PERFORMANCE EVALUATION


FACILITATIVE ADMINISTRATIVE

SUPPORTS


SUPPORTS

RECRUITMENT AND SELECTION


PRESERVICE TRAINING

PRESERVICE TRAINING

SYSTEMSINTERVENTIONS


Implementation Drivers

DECISION SUPPORT DATA SYSTEMS



Teaching Interaction Components

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

N = 7 N = 6 N = 3 N = 7 N = 8 N = 9

A B C D E F

Workshops

Perc

en

t o

f C

om

po

nen

ts

Dem

on

str

ate

d

PRE

POST

OUTCOMES(% of Participants who Demonstrate Knowledge, Demonstrate

new Skills in a Training Setting, and Use new Skills in the Classroom)

TRAININGCOMPONENTS

KnowledgeSkill

DemonstrationUse in the Classroom

Theory and Discussion

10%

5% 0%

..+Demonstration in Training

30%20%

0%

…+ Practice & Feedback in Training

60% 60% 5%

…+ Coaching in Classroom

95% 95% 95%

Joyce and Showers, 2002


Keys to CoachingCoach with expertise in the innovation

Coaching skills (how to: observe & describe behavior; give & take feedback; do behavior rehearsal; shape & reinforce competence)

Coaching content integrated with selection, training, and staff performance assessment content

Coaching delivery plan (coach: practitioner ratio; frequency and nature of contact with the practitioner)


Keys to CoachingMultiple sources of information about each practitioner (weekly direct observation; data/ information reviews)

Coaching content related to craft knowledge & personal style (innovation – craft – person fit)

Staff development plan (frequently updated)

360 – degree feedback


Implement Innovations

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION

INT

ER

VE

NT

ION Perf. Assess.

Fidelity

Implementation without fidelity does not work


Functional Fidelity Assessment

0%

10%

20%

30%

40%

50%

Homes

1.0

1.2

1.5

1.7

1.9

2.0

1.1

1.4

1.6

1.8

1.3

Delinquency

Teaching

rs = – .94

Pe

rce

nt

Pa

ren

tal-

Tea

ch

ing

Wit

h Y

ou

ths

Me

an

Se

lf Rep

orte

dD

elin

qu

en

cy

Ratin

gs

1 2 3 4 5 6

Bedlington, et al., 1988


Functional Fidelity Assessment

0%

10%

20%

30%

40%

50%

Homes

5.0

5.4

6.0

6.4

6.8

7.0

5.2

5.8

6.2

6.6

5.6

Youth Ratings

Teaching

rs = .94

Pe

rce

nt

Pa

ren

tal-

Tea

ch

ing

Wit

h Y

ou

ths

Me

an

Yo

uth

Sa

tisfac

tion

Ra

ting

s

1 2 3 4 5 6 7 8 9 10

Bedlington, et al., 1988


INTEGRATED & COMPENSATORY






SUPPORTS


SUPPORTS



PRESERVICE TRAINING

PRESERVICE TRAINING



Implementation Drivers




Implementation

Who does the work?


Implementation Team

A group that knows the innovation very well (formal and craft knowledge)

A group that knows how to implement that innovation with fidelity and good effect

A group that accumulates data & experiential knowledge -- more effective and efficient over time (information economics, K. Arrow)


Implementation Team

School

Management (leadership, policy)

Administration (HR, structure)

Supervision (nature, content)

Teacher

State and Community Context

District

Imp

lem

enta

tio

n T

eam

Simultaneous, Multi-Level Interventions


Implementation TeamChange the behavior of adult education professionals

“Systems don’t change, people do” (J.W.)

Change organizational structures, cultures, and climates

Change the thinking of system directors and policy makers

Successful and sustainable implementation of evidence-based programs always requires organization and systems change.

0.0

1.0

2.0

3.0

4.0

7/7

7

7/7

8

7/7

9

7/8

0

7/8

1

7/8

2

7/8

3

7/8

4

N = 41 Group Homes (152 Couples)

Tra

inin

g L

eve

l

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

Eff

ect

ive

ne

ss

Training Outcome

Implementation & Outcome

Fixsen, Phillips, et al. (1985)

r = .95


Costs and Savings

Implementation Costs & Savings(Inflation Adjusted)

60

70

80

90

100

110

120

1 Yr Pre During PostYear 1

PostYear 2

PostYear 3

Ch

ang

e in

Bu

dg

et (

Per

cen

t)


Budget Changes (4 years)

Pre Post

Administration 39% 19%

Treatment Units 59% 75%

Training & Eval. 2% 6%

Reduction in administrative staff, reduction in maintenance staff and repairs, reduction in food/staff costs, nearly 100% staff redeployment/rehiring into newly defined positions


“Making it happen”

Implementation: Active involvement of implementation teams that work at the intersection of practices, programs, systems, communities, & scientists

Implementation teams are accountable for assuring use of innovations with fidelity and good outcomes



Implementation Team

Implementation Team

Prepare Communities

Prepare schools faculty, staff

Work with Researchers

Assure Implementation

Prepare Districts Assure Student Benefits


ORG

S

T

CPE

DSDS

FA

SI

Developer - IT

Developer

IT

IT

IT

IT

IT - Developer

Most Developers

IT = Implementation Team


ORG

S

T

C

PE

DSDS

FA

SI

DeveloperDeveloper

Developer

IT

IT

IT

IT

Some Developers


ORG

S

T

C

PE

DSDS

FA

SI

IT

Sustainability Stage


Systems Change

“Systems trump programs.”Patrick McCarthy, Annie E. Casey Foundation


Systems Change

Teachers / staff impact children

It is the job of administrators, managers, and funders to align policies and structures to facilitate effective teacher / staff practices

There is no such thing as an “administrative decision” – they are all education decisions


Systems Change

New practices do not fare well in old organizational structures and systems

Develop new position descriptions and job functions in state departments and districts focused on implementation (effective use) of policies and innovations


Federal Spending on K-12 Education under the Elementary and Secondary Education Act and NAEP Reading Scores (Age 9)


A Sobering Observation

"All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get."

R. Spencer DarlingBusiness Expert


Systems Change

State Department

District

Schools

Teachers / Staff

Effective Practices

AL

IGN

ME

NT

Federal Departments

Imp

lem

enta

tio

n T

eam

s


Systems Change

Policy (Plan)

Practice (Do)

Structure

Procedure

Policy

Practice

Form follows Function

Imp

lem

enta

tio

n

Tea

m

Stu

dy - A

ct

OUTCOMES


Creating Capacity for Competent Change

State Implementation and Scaling up of Evidence-based Practices

Capacity = knowledge, skills, and experience; self-correcting, self-sustaining knowledge utilization

State Management

Group

State Transformation

Team

Regional Implementation

Team

N = 50 – 200 Schools


Team



Team



Team


IMPLEMENTATION CAPACITY FOR

SCALING UP EBPs

Intensive Development

Saturation

State Capacity Development

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12

Years

Reg

iona

l Im

pl. T

eam

s RITs

STTs

Intensive Development

Saturation44/51


Outcomes for Children, Families,

& Communities

Functional Education

Bureaucracy

Teachers

Policies

Schools


SISEP State Implementation and Scaling up

of Evidence-based Practices

www.scalingup.org“Just for States”

Selection criteria/ rationales

“Resources”

Concept paper

Annotated bibliography


Thank YouWe thank the following for their support

Annie E. Casey Foundation (EBPs and cultural competence)

William T. Grant Foundation (implementation literature review)

Substance Abuse and Mental Health Services Administration (implementation strategies grants; NREPP reviews; SOC analyses of implementation; national implementation awards)

Centers for Disease Control & Prevention (implementation research contract)

National Institute of Mental Health (research and training grants)

Juvenile Justice and Delinquency Prevention (program development and evaluation grants

Office of Special Education Programs (Capacity Development Center contract)

Agency for Children and Families (Child Welfare Leadership Development contract)


For More Information

Dean L. Fixsen

813-974-4446

[email protected]

Karen A. Blase

813-974-4463

[email protected]

National Implementation Research Network

At the Louis de la Parte Florida Mental Health Institute

University of South Florida

http://nirn.fmhi.usf.edu


For More Information

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:http://nirn.fmhi.usf.edu/resources/publications/Monograph/index.cfm

Implementation Research: A Synthesis of the Literature

dean l. fixsen, karen a. blase, michelle duda, sandra naoom, melissa van dyke, frances wallace...

Documents