Quantitative Analyses of the Effectiveness of a Newly

Designed Preceptor Workshop

VA Palo Alto Health Care SystemLilly Liu BMed, RN, CCRN, RN-BC

Denise Fillipucci PhD, RN, GCNS

Satish Mahajan PhD, MStat, MEng, RN

VA Palo Alto Health Care System

Covers 3 campuses- Palo Alto, Menlo Park, and Livermore serving more than 85,000 veterans

Operates >800 beds, 3 extended care facilities, and 100-bed Homeless Domiciliary

Home of several Regional Treatment Centers- Polytrauma Rehabilitation/Brain Injury, Spinal Cord Injury, Western Blind Rehabilitation Center, Geriatric Research Education and Clinical Center, National Center for PTSDTeaching hospital serving 29 schools of nursing and Stanford University School of Medicine

Approximately 1800 nursing staff members

Background

1- day Preceptor Workshop first implemented in 2010

• Adult learning theory

• Communication

• Giving feedback

• Learner evaluation

Internal data suggested important gaps existed

• Knowledge in areas of preceptor roles/responsibilities

• Teaching in clinical settings

• Providing support to learners

• Structuring orientations

• Critical thinking & evaluation

• Giving corrective feedback

• Managing challenging behaviors

Literature Review

Precepting perceived as stressful internationally

Preceptors lack educational preparation

Role ambiguity

Role overload

Lack of teaching time

Sense of added responsibilities

Lack of leadership support

Lack of recognition

(Hautala, Saylor, & O’Leary-Kelly, 2007; Panzavecchia & Pearce, 2014)

Literature Review

Formal preceptor programs elements

Educational preparation

Ongoing education

Support for preceptors

Delineation of preceptor roles and responsibilities

Specification of preceptor selection criteria

(L’Ecuyer, Hyde, & Shatto (2018); Ward & McComb (2017))

Literature Review

Learning Needs of Preceptors

Foy, Carlson & White (2013) Top 5 needs - feedback, critical thinking, prioritization, organization, expectations of preceptor

Johnstone & Mohide (2009) Addressing diversity in clinical education & preceptors’ understanding of own beliefs/values of diversity

Benston & Carlson(2015); Chang, Lin, Chen, Kang & Chang (2015); L’Ecuyer et al (2018); Tsai et al (2018)

Communication, teaching, learning strategies, creating caring learning environment, drafting clinical education plans, evaluating preceptees, managing stress and conflict

Literature Review

Best Practices for Program Design

Senyk & Staffileno (2017) Online learning or blended online and live classes -expand accessibility

Wilson et al. (2013) Simulation as an adjunct method - more helpful than lecture alone

Cotter & Dienemann(2016)

Create opportunities for networking and discussion among preceptors

Literature Review

ProgramEvaluation

Quantitative and qualitative measures are used in program evaluation

Literature review showed a lack of interventional studies designed with valid and reliable assessment tools

None with control groups showing support of preceptor development

Most studies addressed learner satisfaction and self-efficacy rather than behavioral change

(Windey et al. (2015))

New Preceptor Workshop Design

Goal of the new workshop

To provide comprehensive evidence-based knowledge and skills to novice and experienced nurse preceptors

Sources of learning needs assessment

Prior 1-day preceptor workshop evaluation results

Preceptors’ anecdotal feedback

Preceptors’ learning needs assessment findings

Literature review findings on preceptor preparation and program evaluation

New Workshop Agenda

Day 1 Day 2

1. Welcome/Introduction 1. Reflection of Day 1 learning/Kahoot

2. Overview of preceptor roles, responsibilities, and qualifications 2. Competency validation and performance evaluation

3. Preceptorship planning and implementation 3. Time management and prioritization

4. Teaching adult clinicians 4. Using evidence-based practice in precepting

5. Critical thinking 5. Managing challenging behaviors in precepting

6. Professional relationship, communication, and feedback 6. Ethical and legal consideration in precepting

7. Conflict resolution 7. Reality shock and transition shock

8. Putting it all together/simulation

Teaching Strategies

Didactic presentation

Case studies

Group discussion

Role-playing

Simulation

Games

Video clip

Comprehensive workbook

Program Implementation

Officially implemented in March 2017

Seven workshops offered in 2017

Audience All new nurse preceptors (RNs and LVNs) Experienced preceptors

Nurse manager’s approval required Inpatient and outpatient settings

Respiratory therapists

Program Evaluation

Kirkpatrick Model of 4 Levels of Evaluation

Level 1 measures the learners’ reaction or satisfaction with the training

Level 2 measures the extent of learning

Level 3 measures the learners’ behavioral change after the training

Level 4 measures tangible long-term outcomes that can be directly related to the training

(Kirkpatrick & Kirkpatrick (2005))

Program Evaluation Questions

What was the effect of a 2-day preceptor workshop on preceptors’ confidence and comfort in various precepting roles?

Sandau et al.’s (2011) preceptor survey instrument

Is there an improvement in participants’ self-reported perception of precepting knowledge, skills, and attitudes before and after attending the workshop?

Self-Assessment tool

How satisfied were the participants in terms of the workshop adding to their knowledge, skills, and attitudes in precepting?

Course evaluation form

Instruments

Sandauet al.’s (2011) Preceptor surveys

Validated instrument: quantitative and qualitative

Baseline surveys and post-workshop surveys

To measure the preceptors’ knowledge, skills, and behavioral changes in precepting

Addressed levels 2-3 of Kirkpatrick model

Instruments

Quantitative 2 sets of questions

5-point Likert scale

To measure preceptor’s confidence and comfort in five preceptor roles

Ability to precept a new nurse

Active coaching in critical thinking

Working with an orientee with a different personality or learning style

Working with an orientee with a different ethnic background

Providing positive and constructive feedback to an orientee

To assess changes in preceptor after the workshop in actively coaching critical thinking and providing feedback to orientee

Sample Questions on Sandau’s Survey Instruments1. How satisfied are you with your previous education regarding precepting an orientee?

2. How confident are you with your ability to precept a new orientee?

3. How comfortable are you in actively coaching critical thinking with your orientee?

4. How comfortable are you in working with an orientee who has a different personality or learning style than yours?

5. How confident are you in working with an orientee of a different ethnic background than yours?

6. How confident are you in providing both positive and constructive feedback to an orientee?

Self-Assessment Tool

Developed by Nursing Professional Development practitioners based on the workshop learning outcomes

5-point Likert scale

To be completed by the participants before the workshop and immediately after the workshop

Eight items to measure self-reported change in knowledge, skills, and attitudes before and after the workshop

Addressed level 2 of Kirkpatrick model

Self-Assessment Questions 1. Describe preceptor roles and responsibilities

2. Explain how to plan and implement preceptorship (learning needs assessment, select proper instructional methods, etc.)

3. Apply adult learning principles in the clinical setting

4. Implement strategies for promoting critical thinking skills

5. Explain how to provide constructive feedback

6. Identify the strategies for conflict resolution

7. Identify the strategies for proper time management, delegation, and managing challenging behaviors

8. Recognize the importance of competency verification and performance evaluation

Course Evaluation Form

Developed by Nursing Professional Development practitioners following the organization’s recommended course evaluation standards

To assess overall value added for the preceptors by attending this workshop

5-point Likert scale addressed level 1 of Kirkpatrick Model

Sample Questions on Course Evaluation Form

1. Overall, I was satisfied with this learning activity

2. The learning activities and/or materials were effective in helping

me learn the content (handouts, simulation, assessment tools, etc.)

3. I learned new knowledge and skills from this learning activity

4. The scope of the learning activity was appropriate and relevant to my professional needs

5. I will be able to apply the knowledge and skills learned to improve my job performance

6. The training environment was effective for my learning

Quantitative Data Analysis: Setup

Workshop (2 days)

Pre-workshopSurvey Instrument (6 Qns.)

Pre-workshopSelf-assessment (8 Qns.)

Post-workshopSurvey Instrument (6 Qns.)

Post-workshopSelf-assessment (8 Qns.)

Post-workshopCourse evaluation (6 Qns.)

Time lapse (4 months)

Cluster sampling method – classroom environment

No identification of participants & anonymity of responses

Likert scale and continuity of measure assumed

Each question treated separately for testing and analysis

7 cohorts of preceptorship class ranging from 6 to 17 students

n=89 pre-workshop samplesn=16 post-workshop samples

Quantitative Data Analysis: Test Assumptions

Test assumptions for each of five datasets separately

1. Power Calculations

• Do we have sufficient data for the set error tolerance and effect size?

• Type I Error 1-5%

• (1-Type II Error) = Power of Test 80-99%

• Effect Size Cohen’s d (measure of effect size)

• Alternate Hypothesis (Ha) > or ≠

• If sufficiently powered small non-response rate is not a major concern

Quantitative Data Analysis: Test Assumptions

2. Normality of datasets

• Many tests are based on normality assumption – what’s that?

• Graphical test: QQ Plot

• Analytical test: Shapiro-Wilks test

Quantitative Data Analysis: ResultsObjective 1: To compare pre- & post-workshop group averages

If the data shows normal distribution use independent/paired t-test

Otherwise use Wilcoxon Rank Sum test

Used Wilcoxon Rank Sum test

Sandau’s Survey Instrument showed improvements in post- workshop ratings when

compared to pre-workshop ones*

Self-assessment tool showed improvements in post-workshop

ratings when compared to pre-workshop ones

Quantitative Data Analysis: ResultsObjective 2: To compare across cohorts

If the data shows normal distribution use one-way ANOVA

Otherwise use Kruskal-Wallis test

Used Kruskal-Wallis test

Pre-workshop Sandau’s Survey Instrument baseline assessment of preceptors

Pre- & post-workshop differences in process of learning for preceptors

Self-assessment tool

Post-workshop Course Evaluation Instructor’s contribution to process of

learning for preceptors

Quantitative Data Analysis: ResultsObjective 3: Notable characteristics of the analysis

How confident are you in working with an orientee of a different ethnic background than yours? Not statistically significant Consistent exposure to diversity in workplace

Behavioral Change: Have you given formal feedback, both positive and constructive? Statistically significant Improved attitude towards providing feedback

Behavioral Change: Have you used techniques to promote critical thinking? Not statistically significant Room for improvement in efforts to promote critical thinking

Quantitative Data Analysis: ResultsObjective 3: Notable characteristics of the analysis

Course Evaluation Lowest Score: “The learning activities and/or materials were effective in helping me learn the content”

Course Evaluation Highest Score: “I learned new knowledge and skills from this learning activity”

Instructors teaching this workshop provided the essential skills and knowledge to the workshop participants but could improve on teaching techniques and course materials

All datasets (both pre & post) compared across all 7 cohorts

Uniformly statistically significant results

Students were uniformly uninformed about preceptorship

Instructors were prepared equally for all the classes and consistently delivered effective workshops throughout the year

Uniform class environment, materials, and take home messages

Limitations

Kirkpatrick Level 4 not addressed

Which long-term outcomes should be considered for program evaluation

Qualitative data analysis remains unaddressed

Implications for Clinical Practice and Future Research

Lack of preceptor preparation

Strategies for teaching preceptors on how to improve critical thinking for orientee

Strategies to improve post-workshop survey response rate

Involve orientee in the evaluation process

Conclusion

Quantitative analyses have shown the effectiveness of our newly designed 2-day preceptor workshop

Integrated quality improvement process with evidence-based practice

Added additional evidence to the literature in enhancing preceptor education

Provided directions for future research

References Bengtsson, M., & Carlson, E. (2015). Knowledge and skills needed to improve as preceptor: Development of a

continuous professional development course—A qualitative study Part I. BMC Nursing, 14(1), 51. doi:10.1186/s12912-015-0103-9

Chang, C., Lin, L., Chen, I., Kang, C. Y., & Chang, W. Y. (2015). Perceptions and experiences of nurse preceptors regarding their training courses: A mixed method study. Nurse Education Today, 35(1), 220–226. doi:10.1016/j.nedt.2014.08.002

Cotter, E., & Dienemann, J. (2016). Professional development of preceptors improves nurse outcomes. Journal for Nurses in Professional Development, 32(4), 192–197. doi:10.1097/ nnd.0000000000000266

Foy, D., Carlson, M., &White, A. (2013). RN preceptor learning needs assessment. Journal for Nurses in Professional Development, 29(2), 64–69. doi:10.1097/nnd.0b013e318287aa12

Hautala, K. T., Saylor, C. R., & O’Leary-Kelley, C. (2007). Nurses’ perceptions of stress and support in the preceptor role. Journal for Nurses in Staff Development, 23(2), 64–70. doi:10.1097/01.nnd. 0000266622.39304.31

Johnston, C., & Mohide, E. A. (2009). Addressing diversity in clinical nursing education: Support for preceptors. Nurse Education in Practice, 9(5), 340–347. doi:10.1016/j.nepr. 2008.08.005

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2005). Transferring learning to behavior: Using the four levels to improve performance. San Francisco, CA: Berrett-Koehler.

References L’Ecuyer, K., Hyde, M., & Shatto, B. (2018). Preceptors’ perception of role competency. The Journal of Continuing Education in

Nursing, 49, 233–240.

Panzavecchia, L., & Pearce, R. (2014). Are preceptors adequately prepared for their role in supporting newly qualified staff? Nurse Education Today, 34(7), 1119–1124. doi:10.1016/j.nedt.2014.03.001

Sandau, K. E., Cheng, L. G., Pan, Z., Gaillard, P. R., & Hammer, L. (2011). Effect of a preceptor education workshop: Part 1. Quantitative results of a hospital-wide study. The Journal of Continuing Education in Nursing, 42(3), 117–126. doi:10.3928/00220124-20101101-01

Senyk, J., & Staffileno, B. A. (2017). Reframing nursing preceptor development. Journal for Nurses in Professional Development, 33(3), 131–137. doi:10.1097/nnd. 0000000000000343

Tsai, Y., Lee-Hsieh, J., Turton, M. A., Li, S., Tseng, H., Lin, H., & Lin, H. (2014). Nurse preceptor training needs assessment:Views of preceptors and new graduate nurses. The Journal of Continuing Education in Nursing, 45(11), 497–505. doi:10.3928/00220124-20141023-01

Ward, A., & McComb, S. (2017). Precepting: A literature review. Journal of Professional Nursing, 33(5), 314–325. doi:10.1016/j.profnurs. 2017.07.007

Wilson, R., Acuna, M., Ast, M., & Bodas, E. (2013). Evaluation of the effectiveness of simulation for preceptor preparation. Journal for Nurses in Professional Development, 29(4), 186–190. doi:10.1097/ nnd.0b013e31829aec2e

Windey, M., Lawrence, C., Guthrie, K.,Weeks, D., Sullo, E., & Chapa, D. W. (2015). A systematic review on interventions supporting preceptor development. Journal for Nurses in Professional Development, 31(6), 312–323. doi:10.1097/nnd.0000000000000213

Thank You! Questions?

VA Palo Alto Health Care SystemLilly Liu BMed, RN, CCRN, RN-BC

Denise Fillipucci PhD, RN, GCNS

Satish Mahajan PhD, MStat, MEng, RN