chris borgmeier, ph.d. michelle duda, ph.d. melissa van dyke, lcsw evidence-based professional...
TRANSCRIPT
Chris Borgmeier, Ph.D.
Michelle Duda, Ph.D.
Melissa Van Dyke, LCSW
Evidence-based Professional Development
2011 SPDG Regional Meeting
A Few Questions: What do we mean when we use the term,
“professional development?”
What outcomes are we attempting to achieve?
What do we know about the professional development strategies that are likely to achieve particular outcomes?
How can we monitor and improve the quality of professional development over time?
Overcoming Barriers with
Common language
Common frameworks
Defined approach (best practices)
Common measures (commitment to continuous quality improvement)
Common PD Challenges
COMMON LANGUAGEOvercoming Barriers :
Training
“the process of bringing a person to an agreed standard of proficiency, by practice and instruction”
http://www.thefreedictionary.com/training
“the acquisition of knowledge, skills, and competencies as a result of the teaching of vocational or practical skills and knowledge that relate to specific useful competencies.”
http://en.wikipedia.org/wiki/Training
Adult Learning Defined
“a collection of theories and methods for describing the conditions under which the processes of learning are optimized (Merriam, 2001; Trotter, 2006; Yang, 2003).”
Trivette, Dunst, Hamby, O’Herin, 2009, p. 1
Professional Development
"Professional development ... goes beyond the term 'training' with its implications of learning skills, and encompasses a definition that includes formal and informal means of helping teachers not only learn new skills but also develop new insights into pedagogy and their own practice, and explore new or advanced understandings of content and resources.”
Modified from http://www.ncrel.org/sdrs/areas/issues/educatrs/profdevl/pd2prof.htm
Professional Development (cont.)
“. . . [This] definition of professional development includes support for teachers as they encounter the challenges that come with putting into practice their evolving understandings about the use of new skills and technology.”
Modified from http://www.ncrel.org/sdrs/areas/issues/educatrs/profdevl/pd2prof.htm
COMMON FRAMEWORKSOvercoming Barriers :
Effective Interventions
Actual SupportsYears 1-3
Outcomes Years 4-5
Every Teacher Trained
Fewer than 50% of the teachers received some training
Fewer than 10% of the schools used the CSR as intended
Every Teacher Continually Supported
Fewer than 25% of those teachers received support
Vast majority of students did not benefit
Aladjem & Borman, 2006; Vernez, Karam, Mariano, & DeMartini, 2006
Longitudinal Studies of a Variety of Comprehensive School Reforms
Implementation Science
What Works
Implementation Drivers Common features of successful
supports to help make full and effective uses of a wide variety of innovations
Core Implementation
Components
Core Implementation
Components
© Fixsen & Blase, 2008
Improved Outcomes for . . .
Com
pete
ncy
Driv
ers
Com
pete
ncy
Driv
ers O
rganization Drivers
Organization D
rivers
LeadershipLeadership
Program/Initiative (set of practices)
How:
What:
Why:
Capacity to provide direction/vision of process
Staff capacity to support students/families with the selected practices
Institutional capacity to support staff in implementing practices with fidelity
Core Implementation
Components
Core Implementation
Components
© Fixsen & Blase, 2008
Improved Outcomes for . . .
Com
pete
ncy
Driv
ers
Com
pete
ncy
Driv
ers O
rganization Drivers
Organization D
rivers
LeadershipLeadership
Program/Initiative (set of practices)
How:
What:
Why:
Capacity to provide direction/vision of process
Institutional capacity to support staff in implementing practices with fidelity
Professional DevelopmentProfessional Development
© Fixsen & Blase, 2008
Performance Assessment (Fidelity)
Coaching
Training
Selection
Systems Intervention
Facilitative Administration
Decision Support Data System
Integrated & Compensatory
Com
pete
ncy
Driv
ers
Com
pete
ncy
Driv
ers O
rganization Drivers
Organization D
rivers
LeadershipLeadership
Adaptive Technical
Improved Outcomes for . . .
Program/Initiative (set of practices)
Why Focus First on Teachers’ Behavior?
In education, the Teacher IS THE INTERVENTION
Build supports in relationship to what the Teacher needs to be competent
Create the conditions under which teachers can do the right thing for the right reason at the right time to maximize positive results
Wide ranging inputs (individuals with all their past history and current realities)
Build alignment of implementation practices and outcomes on what needs to happen at the point of the “learning exchange”
Improved outcomes for students
Participants with improved knowledge and skills around
identified practice(s)
• Think about those who are the participants in (receivers of) your professional development activities…
• What features would you suggest be in place around the Competency, Organization and Leadership drivers so that professional development is provided in a “host environment” to support sustainability and promote fidelity?
Participants of Professional Development
Participants with improved knowledge and skills around
identified practice(s)
Professional development providers
• Think about those who are the providers of your professional development activities…
• What features would you suggest be in place around the Competency, Organization and Leadership drivers so that your providers conduct quality professional development?
Providers of Professional Development
DEFINED APPROACHOvercoming Barriers:
“Best Practices”
The BIG QUESTION:
What outcomes are you hoping to
achieve?
Use training “alone” strategies if…
You want to disseminate new information
You want to create “buy in”
You want to clarify “truths” and dispel “myths”
You are talking with a broad audience
Use training “alone” strategies …
To increase knowledge about the effective program or practice related to. . . underlying theory of change intervention components rationales of key practices
To increase familiarity with the use of new skills
To increase awareness of what it feels like to begin to use new skills and to receive expert feedback
Training “Best Practices”
Theory grounded (adult learning)
Skill-based
Data-based (pre and post testing)
Feedback to Selection and Feed Forward to Supervision
Trainers have been trained and coached
Training “Best Practices”
Theory grounded (adult learning)
Skill-based
Feedback to Selection and Feed Forward to Supervision
Data-based (pre and post testing)
Trainers have been trained and coached
What do we know about each
of these “best practices?”
Adult Learning “Best Practices”
The most effective training includes learner experiences related to planning, application, and deep understanding
Use a diverse array of Adult Learning methods (“where 5 or 6 adult learning method characteristics were used, the average effect size was almost 1.25”)
Learners need to be engaged actively in the learning process
Multiple learning experiences, large doses of learner self-assessment and reflection, instructor facilitated learner assessment
Small numbers of participants, multiple occasions
Trivette, Dunst, Hamby, O’Herin, 2009, p. 10 -11
Adult Learning “Best Practices”
Trivette, Dunst, Hamby, O’Herin found best practices in each of the following categories:
Introducing Information
Illustrate/demonstrate
Practicing
Evaluation
Reflection
Mastery
Trivette, Dunst, Hamby, O’Herin, 2009, p. 6 - 8
Skill-based “Best Practices”
Behavior Rehearsals (vs. Role Plays)
Knowledgeable Feedback Providers
Practice to Criteria
Data Based “Best Practices”
Develop and use pre/post tests to determine to what extent knowledge and skill levels are being improved
Outcome data collected and analyzed
Fidelity measures collected and analyzed
Training
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
N = 7 N = 6 N = 3 N = 7 N = 8 N = 9
A B C D E F
Workshops
Pe
rce
nt
of
TI C
om
po
ne
nts
De
mo
ns
tra
ted
PRE
POST
Collins, S. R., Brooks, L.E., Daly, D.L., Fixsen, D.L., Maloney, D.M., & Blase, K. A. (1976)
Staff Training
Feedback “Best Practices”
Feed Forward of pre/post data to Coaches/Supervisors
Feedback of pre/post data to Selection and Recruitment
Feedback “Best Practices”
Of course, there are even more opportunities to benefit from feedback…
© Fixsen & Blase, 2008
Performance Assessment (Fidelity)
Coaching
Training
Selection
Systems Intervention
Facilitative Administration
Decision Support Data System
Integrated & Compensatory
Com
pete
ncy
Driv
ers
Com
pete
ncy
Driv
ers O
rganization Drivers
Organization D
rivers
LeadershipLeadership
Adaptive Technical
Improved Outcomes for training participants
Effective Training Practices
Implementation-informed PD
If Professional Development is in support of a well-defined, effective intervention. . .
. . . then Professional Development includes
Training
Coaching
Performance assessment
and Selection
Training, plus coaching
Some well-defined interventions (EBPs) will require additional support to teachers post-training: Joyce and Showers (2002) data
» With coaching = 95% use in classrooms
» Without = 5% use in classrooms
Rogers, Wellens, & Conner (2002) data
» About 10% of what is taught in business workshops actually is put into practice
Significant data across domains that feedback improves performance
Coaching “Best Practices”
Design a Coaching Service Delivery Plan Use multiple sources of information for
feedback – Direct observation is critical Provide regular feedback to all “Drivers” Develop accountability structures for Coaching
– Coach the Coaches! Regular review of adherence to Coaching Service
Delivery Plan Multiple sources of information for supervisor
feedback
Performance Assessment Best Practices
Transparent Processes – Orientation
What, When, How, Why
Use of Multiple Data Sources
Context
Content
Competency
Tied to positive recognition – not used ‘punitively’
Selection “Best Practices”
Job description clarity about accountability and expectations
Pre-Requisites are related to “new practices” and expectations (e.g. basic group management skills).
Interactive Interview Process:
Behavioral vignettes and Behavior Rehearsals
Assessment of ability to accept feedback
Assessment of ability to change own behavior
Enhancing Professional Development
Promotes adult learning and changes adult behavior
Empowers individuals to improve their craft
Ensures that new skills are used and/or current skills are refined
Integrates selection, training, coaching, and performance assessment to promote the development of a highly effective workforce
Supports the creation of a “hospitable environment” to allow for these new behaviors to be used and improved
Core Implementation
Components
Core Implementation
Components
© Fixsen & Blase, 2008
Improved Outcomes for . . .
Com
pete
ncy
Driv
ers
Com
pete
ncy
Driv
ers O
rganization Drivers
Organization D
rivers
LeadershipLeadership
Program/Initiative (set of practices)
How:
What:
Why:
Capacity to provide direction/vision of process
Institutional capacity to support staff in implementing practices with fidelity
Professional DevelopmentProfessional Development
Discussion Question
To what extent are current approaches
to professional development
designed to develop the necessary
competence in participants to
skillfully use the most effective
educational approaches to
accomplish the goal of improved
outcomes for students?
COMMON MEASURESOvercoming Barriers :
“Commitment to Continuous Quality Improvement”
Integrating All We Know
A well-defined and effective intervention
Clarity about the “active ingredients”
Clear criteria to assess performance (that correlates with the desired outcomes)
An effective approach to training
An effective coaching system
An effective performance assessment process
Use of selection “best practices”
Improvement Cycles
Improving PD Effectiveness
Collect and use data (pre/post tests to determine to what extent knowledge and skill levels are being improved)
Develop the capacity of the trainers (establish clear performance criteria, based on “what works,” then select, train, coach, and assess trainers to criteria)
Improve the ways in which training participants are “prepared” for training (create readiness)
What did we learn?
What do we mean when we use the term, “professional development?”
What outcomes are we attempting to achieve?
What do we know about the professional development strategies that are likely to achieve particular outcomes?
How can we monitor and improve the quality of professional development over time?
For More Information
Melissa Van [email protected]
Michelle Duda [email protected]
Chris [email protected]
For More InformationState Implementation and Scaling up of
Evidence-based Practices (SISEP)
Dean Fixsen, Karen Blase, Rob Horner, George Sugai
www.scalingup.org
Concept paper
Annotated bibliography
Data on scaling up
Scaling up Briefs
www.implementationconference.org
Evidence-based
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).
Download all or part of the monograph at:
http://www.fpg.unc.edu/~nirn/resources/detail.cfm?resourceID=31
Implementation Research: A Synthesis of the Literature
Thank You for your Support
Annie E. Casey Foundation (EBPs and cultural competence)
William T. Grant Foundation (implementation literature review)
Substance Abuse and Mental Health Services Administration (implementation strategies grants; national implementation awards)
Centers for Disease Control & Prevention (implementation research)
National Institute of Mental Health (research and training grants)
Juvenile Justice and Delinquency Prevention (program development and evaluation grants
Office of Special Education Programs (scaling up capacity development center)
Administration for Children and Families (child welfare reform; capacity development)
Duke Endowment (child welfare reform)