28 Healthcare Management Forum Gestion des soins de santé

he Preceptor & Mentor Initiative for Health Sciences in BC ProjectTeam developed an evaluation plan during the initial programplanning phase in 2001. To bring objectivity to this criticalfunction, the evaluation framework was designed with inputfrom, but independent of, the Project Team. We neededrepresentative evidence from health and education professionalsin BC to assess whether the following objectives of the Initiativehad been met:

• to increase recruitment and retention of health sciencesprofessionals;

• to build capacity to educate these same professionals;

• to contribute to a culture of lifelong learning; and

• to address the needs of many diverse groups.

In addition, the evaluation was designed to elicit recommendations on futurepreceptoring and mentoring structures and activities in the province.

The evaluation approach was based on the quality movement [structure-process-outcome] and aligned with a logic model. When translated intorecommendations, these findings become a decision-making tool for policymakers to determine the Initiative’s value. We expect that such a program’sfuture depends not only on its performance but also on whether the programgoals remain desirable given competing priorities.1

Evaluation Framework/ProcessOur evaluation framework, a Program Logic Model of Preceptor & MentorInitiative, was designed by a doctoral student at UBC’s Centre for HealthServices and Policy Research (see figure 1). Although the long-term outcomesare complex and difficult to measure, the Logic Model was designed toprovide clear and explicit criteria for success as indicated by the short-termoutcomes. The literature indicates the short-term outcomes identified

BRIEF REPORT

AbstractPart I of this two part series, which was published in the Spring 2003 issue ofHealthcare Management FORUM, described the Preceptor & Mentor Initiative forHealth Sciences in BC, a provincial strategy to facilitate preceptoring andmentoring. Activities included creation of an educational web site and a UBCContinuing Studies preceptor course, and provincial workshops on preceptoringand mentoring. The purpose of this second article is to outline the philosophy,framework, results and interpretation of the Preceptor & Mentor Initiative’sevaluation component.

T

Going Provincial: Preceptoring andMentoring Revisited in the Health SciencesPart II– The Evaluation by Carol Mitchell and Brenda Sawatzky-Girling

Carol Mitchell, MSN,was the Project Managerof the Preceptor & MentorInitiative for HealthSciences in BC,Vancouver.

Brenda Sawatzky-Girling, MHA, is ahealthcare policy andmanagement consultant inVancouver.

Healthcare Management Forum Gestion des soins de santé 29

(broader implementation of preceptor and mentor

programs, and more standard implementation of best

practices) correlate with the Initiative goals.2

Each component in the Logic Model contributed to a

Framework for Evaluation Implementation that identified

research questions, performance indicators, success

level or target dates, and how data would be collected.

With this information, we could make process-output

linkages. We anticipated early on that data collection

would be a challenge because this type of database

does not exist provincially, within regions or

organizations and only minimally at the department

level. The sources of data we used are described below

and are illustrated in figure 2.

More standardimplementation ofbest practices

Contribution to culture ofcontinuing learning

Figure 1: PROGRAM LOGIC MODEL OF PRECEPTOR & MENTOR INITIATIVE

Component Process Output Outcome OutcomeShort-term Long-term

Needs Baseline assessment Outline of current status of Contribution to increasedassessment survey programs in the province recruitment and retention

of nurses and healthConsultation with Outline of needs of stakeholders sciences professionalsstakeholders in developing programs

BroaderBaseline assessment Outline of theoretical concepts implementation ofsurvey; collecting Outline of best practices preceptor andexisting resources mentor programs

Compilation and Guidelines for managing,development of Literature review planning, implementing andguidelines and evaluating programs that are resources sensitive to diverse needs

(incl. reward and recognition processes and estimated costs)

Contribution to increased Consultation with Clearinghouse of existing capacity for student stakeholders tools and resources placement

Web site Web site and usersDissemination of Workshops to Workshops and participantsinformation showcase the initiative

Provincial and national Conferences and participantsconferences

Networking Web site Interactiveof users discussion forum Web site and users

Sustaining Developing business Funding application Secure fundsinitiative plan and applying

for funding

Support preceptors and mentors in order to:Increase recruitment of health sciences professionalsIncrease capacity to educate health sciences professionalsIncrease retention of health sciences professionalsFoster a culture of continuous learningAddress the needs of diverse groups

Compilation & Development of Guidelines, Resources

Preceptor & Mentor Initiative Web site

Resource Team

Workshops

Provincial Infrastructure & Coordination

Surveys – Needs Assessment, Follow-up

From Practitioner to Preceptor Course

New

Knowledge

DisseminatingKnowledge

Conferences

Journals

ResearchBroader implementation of preceptor & mentor programsMore standard implementation of best practices

Short Term

Outcomes

Long Term

Outcomes

3

2

1

4

Figure 2: Knowledge Management in Action

30 Healthcare Management Forum Gestion des soins de santé

❶ Surveys – We sent an Internet-based

baseline survey to BC’s health and

education stakeholders in September

2001. A follow-up survey was conducted

in February 2003 to compare the two

points in time.

❷ Stakeholder Consultations – Our

Resource Team of preceptor/mentor

experts and end-users grew to include

221 people from around BC. We held 15

workshops (12 of these outside the

Vancouver area) to help participants

identify what resources they already had

and to gather feedback as to whether our

activities and plans were meeting their

needs. We used this information to

continually monitor and refine future

workshops and other program activities

and outputs.

❸ Web site – The evidence-based

Preceptor & Mentor Initiative web site,

launched in January 2002, was tracked in

terms of utilization and feedback.

“We need a ‘one minute preceptor course’ - we

have little time for courses!” (Workshop

Participant, Prince George BC, February

2002).

❹ Preceptor Course – We developed a

preceptor course in a workshop/on-line

learning format with UBC Continuing

Studies. The From Practitioner to Preceptor

course, launched in January 2003, was

developed from our web site and the

Children’s & Women’s Health Centre of

BC preceptor education program. The

C&W Program had undergone an extensive

evaluation in 2002. Participants provided

feedback on the course itself and the

Initiative web site.

ResultsOverall, the activities of the Preceptor &

Mentor Initiative proved effective in

achieving the desired outcomes. The

following describes how we know various

Initiative activities directly achieved

broader implementation of preceptor

and mentor programs in BC and more

standard implementation of best practices:

❶ Surveys

Twenty-three out of 66 (35%) follow-up

surveys were completed and returned.

Respondents were representative of both

health and education sectors and diverse

in terms of discipline. In comparing the

two points in time between baseline and

follow-up surveys (see figure 3), trends

indicate that respondents were further

along in terms of planning, implementing

and evaluating preceptor and mentor

programs, notably preceptor programs.

❷ Stakeholder Consultations

Workshop evaluation forms and informal

comments were overwhelmingly posi-

tive.The follow-up survey asked whether

respondents had attended or arranged

workshops in their areas (15/23

respondents replied yes). Of these 15,

100% found the workshops helpful in

developing programs, establishing crucial

networks and identifying resources in

their areas.

❸ Web site

Web site utilization rose steadily since its

launch. Web site feedback was systemati-

cally collected and the web site revised

based on this input:

• Over 1,400 visits were made to theweb site with the most intense usein western Canada. A significantnumber of users visited the web siteseveral times.

• On-line feedback forms wereminimal, but all were positive.

• Eighteen out of 23 follow-up surveyrespondents had accessed the website; 17/18 of this group found theweb site to be useful to them inimplementing preceptor/mentorprograms and best practices.

• Users who had web site demonstra-tions in workshops or who used theweb site in the preceptor courseprovided positive comments on howcomprehensive and useful the website was, and that it met their needs.

❹ Preceptor Course

This introductory course directly achievesthe outcomes of broader implementationof preceptor programs and more standardimplementation of best practices. Thereis demand for the course from differentregions of BC. Feedback from thestudents in the initial course offeringregarding quality and usefulness wasconsistently positive.

Figure 3: Pre & Post: State of Preceptor and Mentor ProgramDevelopment in BC

Healthcare Management Forum Gestion des soins de santé 31

Interpretation – Lessons Learned

The Preceptor & Mentor Initiative unfolded

amidst the many system challenges with

which healthcare managers are undoubt-

edly very familiar. Key challenges we

experienced in rolling out this program

included: lack of data on the current state

of preceptoring and mentoring; different

systems that are huge, complex and not

well interconnected; tremendous changes

in terms of people, organizations and

infrastructure that made it difficult to

target leaders and communicate (for

example, in 2002 the BC health system

reorganized from over 50 to six health

authorities); and inherent inequities in

the various systems in terms of employ-

ment contracts, targeted funding and

discipline-specific roles and boundaries.

In 2001, the BC Ministry of Health

Planning recognized the need for a

coordinated approach to preceptoring

and mentoring that bridged the health

and education sectors. The resulting

model was a small infrastructure that

compiled and developed supportive and

knowledge-based resources while

coordinating and working with diverse

communities to facilitate their develop-

ment. Our experience with the Preceptor &

Mentor Initiative and the results it yielded

indicate that such a model can make

sustainable changes at the local level.

If we were to do this project again, with

identical system challenges, we would:

• develop the From Practitioner to

Preceptor course earlier;

• target and assist the less organized

communities of BC earlier; and

• streamline the web site content

management process.

RecommendationsThe Preceptor & Mentor Initiative’s majorrecommendations include:

Continue to:

• value and pursue the goals of theInitiative;

• update the Preceptor & Mentor website as a provincial resource,emphasize on-line networkinggroups;

• support preceptor/mentor educationfor all health disciplines throughoutthe province.

Establish:

• a provincial competency-basedmentoring system for healthcareprofessionals and managers;

• a provincial database onpreceptoring and mentoring (HSPnet3

may achieve this).

"A workplace needs to have ‘mentoring’ as anorm." (Workshop Participant, VancouverBC, January 2002).

AcknowledgementsThe authors would like to thank Alice Chen, (PhD Candidate atUBC’s Department of Health Care & Epidemiology), for sharing herexpertise and developing the Program Logic Model of Preceptor & MentorInitiative and supporting frameworks. Our sincere thanks also go tothe BC Academic Health Council (in particular Dr. John Gilbert andKathy Copeman-Stewart) for their continuous support.

References and Notes1. Grembowski D. The Practice of Health Program Evaluation.

Thousand Oaks, CA: Sage Publications; 2001.

2. References of studies linking aspects of preceptoring and

mentoring with outcomes are available on the Preceptor &

Mentor Initiative web page at:

http://www.couth.bc.ca/managingprograms/evaluating/how

to.htm

3. HSPnet (Health Services Placement Network of BC) is a

province-wide web-enabled database for coordinating and

improving student placements for all professions in BC.

Visit the project’s web site at: http://www.hspbc.net

Opcom Healthcare1/4 page (repeat)