evaluating educational interventions paul stevenson
TRANSCRIPT
![Page 1: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/1.jpg)
Evaluating Educational Interventions
Paul Stevenson
![Page 2: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/2.jpg)
Background
• Local evaluation of EBP training showed:Skills & Knowledge improved ….
..but limited change in behaviour.
•Miller, G E The assessment of clinical skills/performance
•Academic medicine, 1990, 65, S63-7
![Page 3: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/3.jpg)
• How can I make the training more effective?
• I had used the Kirkpartick Model for Evaluation
![Page 4: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/4.jpg)
Kirkpatrick Model – Learner Focused
Level 1 – Learner reaction
Level 2 – Modification of learner attitudes & perceptions,
and learner acquisition of knowledge & skills
Level 3 - Change in learner behaviour
Level 4 - Benefits to the organisation/patient resulting from learner performance
Kirkpatrick’s four levels of evaluation (1967,1994), adapted by Barr et al (2000)
Level 1 – Learner reaction
Level 2 – Modification of learner attitudes & perceptions,
and learner acquisition of knowledge & skills
Level 3 - Change in learner behaviour
Level 4 - Benefits to the organisation/patient resulting from learner performance
Kirkpatrick Model
![Page 5: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/5.jpg)
Kirkpatrick ModelTendency for declarative outcome measures
Shows if the intervention has been effective……but provides little information about why
Learner FocusedFails to consider contextual/environmental variablesView that training is solely responsible for any outcome
One DirectionalTendency to focus on the impact of the learner on their environment
![Page 6: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/6.jpg)
• What alternative methods could be used?
![Page 7: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/7.jpg)
Transfer of LearningBaldwin & Ford (1988) transfer of training into practice. Three areas of influence on transfer :
1. Aspects of the course
2. Characteristics of the learner
3. Features of the work environment
![Page 8: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/8.jpg)
Common Barriers
• The most commonly mentioned barriers to EBP are:
Learner
•Personal lack of interest
•Lack of skills
•Low self-efficacy
•Previous experience
Training
•Appropriate content
•Delivery method
Environment
•Organisational culture
•Insufficient time
•Lack of resources
•Lack of support from colleagues and management
Salbach et al, 2007; Pollock & Legg, 2000; Jette et al, 2003; McColl & Smith, 1998; Freeman & Sweeny,2001; Retsas, 2000; Cranney & Warren, 2001; Metcalf et al, 2001
![Page 9: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/9.jpg)
Diffusion of Innovation
Diffusion of innovation model, Rogers (1995)
![Page 10: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/10.jpg)
LET evaluation
Lewin’s (1951) equation B=f (P,E) Behavior is a function of the person and the environment
Adapting this to evaluate the impact of an educational intervention we get: BC= f (L,E,T)
Behavioral change is a function of:L – Learner E – Environment T- Training
Rather than focus solely on the impact of the learner on the environment, we also measure the impact of the environment on the learner.
![Page 11: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/11.jpg)
Systems Approach
• System: Integrated whole composed of several parts
• Look at the educational intervention holistically
• Look for relationships with other parts of the organisation (L&D, HR, Clinical Audit)
![Page 12: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/12.jpg)
Example Measures
Learner:Pre-post course evaluation formTest exercise/paper to show learning has occurredLikert/VAS measure of self-efficacy, desire for change and other psychological aspects of the learner.
Post-course follow up – has learning been retained, examples of how training has impacted on work, reasons/barriers if new learning has not been used.
Monitor usage statistics such as Athens Usage
![Page 13: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/13.jpg)
Example MeasuresEnvironment:
Floor Walking – observe the working environment: is there access to appropriate resources, space/time for staff to implement EBP.
Dialog with Line Management – Creates a two way exchange of information
Integrate Library Services with the wider organisation – If you are working in the same environment as your learners you will experience the culture/environment first-hand.
![Page 14: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/14.jpg)
Example Measures
Training:Pre-post course evaluation formTest exercise/paper to show learning has occurred
Post-course follow up – has learning been retained, examples of how training has impacted on work.
Peer-observation
![Page 15: Evaluating Educational Interventions Paul Stevenson](https://reader036.vdocuments.site/reader036/viewer/2022062308/56649e2f5503460f94b1f5f8/html5/thumbnails/15.jpg)
Advantages
Truly understand the wider impact of the training
Identify and have a better understanding of the barriers which exist and can start to resolve them.
Increases profile and importance of library services within organisation / departments