innovation in medical education

Innovation in Residency Education

Author: M. Ladhani

Objectives:

•  Identify common problems faced by program directors

and potential innovative solutions.

•  Describe examples of innovations in medical education from the pediatric residency program at McMaster University.

3

Problem 1: Attendance at AHD

5

Mandatory Group Learning Activities (MGLA): Getting Residents Ready for Lifelong Learning.

Author: Moyez Ladhani, Saleem Razack

Background:

•  The Royal College of Physicians of Canada’s CanMEDS project defines the roles of a physician to include seven important competencies

•  Postgraduate training programs must incorporate the teaching and evaluation of the CanMEDS roles.

7

Mandatory Group Learning Activities (MGLA): Getting Residents Ready for Lifelong Learning. | Moyez Ladhani, Saleem Razack

Background

•  Our curriculum for our residents includes mandatory and other teaching sessions.

•  There is a large resource of formal didactic, interactive and case based sessions available for residents to meet their learning objectives.

8


However..

Faculty: “Moyez…I was at AHD and the attendance was poor, a lot of effort went into my talk and if residents aren’t going to show up…etc.” Residents: “ I was taking the time to get some personal matters attended to..” Banana Republic had their in-store only 40% sale that I couldn’t miss….”

9


Background

10


However there was no accountability for residents’ attendance at the numerous activities available to them.

Scholar Role

•  An important aspect of the Scholar role is that: 1.  Physicians are expected to maintain and

enhance professional activities through ongoing learning

2.  Describe the principals of maintenance of

competence, and 3.  Document their own learning processes, to

name a few.

11


Manager:

•  The Manager role expects physicians to implement processes to ensure personal practice improvement and balance patient care, practice requirements and outside life.

12


RCPSC

•  In specialty practice beyond postgraduate training, the RCPSC expects its fellows to participate in the Maintenance of Certification (MOC) process for maintenance and renewal of their fellowship.1

•  We also know that educational meetings improve professional practice and healthcare outcome for patients.2

1.  Frank, JR (Ed). 2005 The CanMEDS 2005 physician competency framework. The Royal College of Physicians and Surgeons Canada.

2.  Forsetlund, L et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2009 Apr 15;(2).

13


The Objectives of the MGLA is to:

•  Develop skills required of them for the Scholar and Manager roles.

•  Maintain and document minimum attendance at

various teaching sessions

14


Getting Ready for MGLA

1.  We identified sessions that should be mandatory and sessions that were optional. Sessions were grouped into 5 categories. For mandatory sessions residents are protected to attend from their clinical duties.

2.  Credits were determined for minimal attendance for each of the session groupings; adjustments were made for year, vacation time, post-call days and time away on electives.

15



3.  Residents were required to document and maintain their attendance. This profile was also maintained in their CanMEDS portfolios.

4.  Failure to maintain their minimal requirements put the resident at risk for promotion.

5.  Residents maintain their activities on the Resident Activity Log

16



17



18



19



20


Reflection is relevant during and after educational activities or clinical encounters. In this way, specialists develop greater

understanding, integrating and translating new knowledge or skills, and in forming future action.3

A Continuing commitment to lifelong learning. Guide to maintenance of certification. The Royal College of Physicians and Surgeons Canada.

Conclusions:

•  This process has not only helped improve attendance but provides a realistic “dry run” for residents with respect to what will be expected of them as future life long learners, Scholars and Managers.

21


Comments for Former Residents

•  “ I liked the system - it did give me some experience in keeping track of learning. Mainly, I found it useful to keep track of my teaching, which along with my notes, helped me with my learning objectives”

•  “The MGLA system was helpful for organization and

recognition of different physician roles…. MGLA prepared me- even just for the concepts- of having to be a manager and scholar”

22


Comments for Former Residents

•  “Keeping lists of hours of educational events attended has some value and is not too time consuming”

•  “ This has helped both as a transition to mainport and,

given that everyone does not attend everything, to have a record that as a resident you have attended a reasonable amount of formal learning (i.e. gives some independence and flexibility to the learning process)”

23


MGLA:

•  Three Take Away Points: 1.  Models the future accountability required of

residents with respect to the ongoing maintenance of certification.

2.  Improved attendance at protected and other teaching activities.

3.  Helps teach the CanMEDS role of Scholar and Manager.

24


Update

•  All our resident now have to be a resident affiliate with the Royal College.

•  This allows them to now log their activities with the

MAINPORT APP

•  As of July 1, 2013, Resident Affiliates who document learning activities in MAINPORT during their residency program can transfer up to 75 credits (25 in each of the MAINPORT categories of Group Learning, Self-Learning and Assessment) into their first five-year Maintenance of Certification (MOC) cycle following certification.

25


Problem 2: Teaching CanMEDS

26

Longitudinal CanMEDS Competencies (LCC): Using experiential learning to implement CanMEDS roles in

post-graduate medical education

Author: Moyez B. Ladhani

Background:

•  The Royal College of Physicians of Canada’s CanMEDS project defines the roles of a physician to include seven important competencies

•  Postgraduate training programs must incorporate the teaching and evaluation of the CanMEDS roles.

28

Longitudinal CanMEDS Competencies (LCC): Using experiential learning to implement CanMEDS roles in post-graduate medical education | Moyez B. Ladhani

The International Conference on Residency Education | La conférence internationale sur la formation des résidents

Background:

•  Dr. Ladhani(PD) •  I have been been struggling with implementing a

curriculum for the so called non-medical expert (NME) roles or intrinsic roles in my pediatric residency program.

29



Background:

•  The Residents: •  “having CanMEDS teaching at AHD takes away from our

medical expert teaching” “Sitting in a large group and listening to a speaker talk on how to collaborate is not useful to my learning”

•  The Literature: •  While NME roles are highly valued, there are challenges

in terms of current strategies for teaching and assessment of these roles.

•  Words such as “frustrating”, “nebulous”, “poorly defined” and “difficult” were used to describe efforts to translate the NME competencies into curriculum

•  Whitehead C, Martin D, Fernandez N, Younker M, Kouz R, Frank J, Boucher A. Integration of CanMEDS Expectations and Outcomes. Members of the FMEC PG consortium; 2011

30



Background:

•  There is strong support for the CanMEDS construct of a ‘good doctor’ as requiring qualities beyond biomedical expertise.

•  The CanMEDS definitions of these NME roles are highly endorsed by both residents and faculty members as appropriately capturing the essential elements of a competent and socially responsible physician.

•  Whitehead C, Martin D, Fernandez N, Younker M, Kouz R, Frank J, Boucher A. Integration of CanMEDS Expectations and Outcomes. Members of the FMEC PG consortium; 2011

31



Experiential Learning

•  is the process of making meaning from direct experience

•  Aristotle once said, "For the things we have to learn before we can do them, we learn by doing them”

32



Experiential Learning:

•  According to David Kolb, knowledge is continuously gained through both personal and environmental experiences. In order to gain genuine knowledge from an experience, certain abilities are required:

1.  the learner must be willing to be actively involved in the experience;

2.  the learner must be able to reflect on the experience;

3.  the learner must possess and use analytical skills to conceptualize the experience; and

4.  the learner must possess decision making and problem solving skills in order to use the new ideas gained from the experience.

33



LCC:

•  The goal of the curriculum was to teach non-medical expert CanMEDS competencies.

•  Residents are taught the multi-facetted Roles they will be

called upon to play in their professional duties

•  The Curriculum covers the 6 intrinsic domains

34



Format:

•  The curriculum is a three year curriculum, one hour every third Tuesday of the month.

•  Residents are in small groups of 11-12 residents ( PGY 1-4) facilitated by two faculty members. “A safe place”

•  Residents are provided advance-reading material and are expected to come to the session prepared for discussion. The sessions involve discussion, role play, video scenarios and occasional lecture format.

35



Format:

•  The facilitators are there to facilitate and are not necessarily content experts. They may or may not lead the discussions.

•  Groups may consider rotating the residents to lead the discussions

36



The Cases:

37



Case Example:

38



Case Example:

39



Case Example:

40



Reflections:

•  Reflection: Residents complete two reflections per year. The residents are expected to share their reflections during sessions scheduled:

1. The residency journey 2. Physician well being

41



Evaluation:

•  There are 4 key sets of skills and behaviours upon which students are evaluated by one another and their facilitators.

•  Students are expected to demonstrate proficiency along all four domains and to continue to maintain/improve over time.

1. Accountability/Respect 2. Respectful Listening 3. Balancing Inquiry and Advocacy 4. Taking Experiential Education Seriously

42



Students Satisfaction:

43


0 10 20 30 40 50 60 70 80 90

100

Strongly Agree

Agree

Somewhat agree


Resident Comments:

•  “Groups allowed for open and honest discussions.” •  “Good mix of medical expert, resident wellness,

communicator” •  “Interactive materials (like on line modules) are good

since they are engaging” •  “I think learning CanMEDS competencies in smaller groups

was more interactive” •  “The one good thing is the actual content. I feel it will be a

better environment to discuss such topics rather than a half-day session”

•  “I feet that this format will help us build skills on how to work as a team”

44



Conclusions:

•  It is important to find ways to help educators and trainees appreciate the intricate associations between the expert role and all other roles.

•  Integration of other roles with that of Medical Expert helps to highlight the fact that the competent physician draws upon various roles simultaneously.

•  Integration of roles teaching and assessment into clinical

contexts gives practical relevance to the roles.

45



LCC:Longitudinal CanMEDS Curriculum

•  3 Take-Away Points 1.  A traditional half day does not allow for discussion and

sharing.

2.  The LCC curriculum uses experiential learning in small group format allowing residents to discuss their experiences

3.  The LCC curriculum helps learners to incorporate the CanMEDS competencies to patient and self-care

46



Discussion

47


Update

•  Residents now suggesting topics and volunteering to write the case scenarios often as part of their teaching block.

48

Problem 3: The Right Shift!

49

Mini-MAS: A Work Based Assessment Tool to Assess Milestones

Author: Moyez B. Ladhani

•  PD •  “I need a better assessment tool to help me assess the

residents”

•  Residents

•  “No one watches me during my day to day work”

51

The International Conference on Residency Education | La Conférence internationale sur la formation des résidents

Work Based Assessment

•  Knowledge, skills and attitudes should be assessed using a multi-faceted longitudinal approach (Sherbino & Frank, 2011).

•  The use of a multi-modal assessment strategy can overcome the limitations of any one-assessment format.

•  Longitudinal assessment monitors ongoing development and avoids excessive testing at any one point (Cox, Irby, &

Epstein, 2007).

52

Mini-MAS/Ladhani


•  Miller (1990) suggests the achievement of competence progresses from “knows” to “knows how” to “shows how” to “does”.

53

Mini-MAS/Ladhani

KNOWS

KNOWS HOW

SHOWS HOW

DOES

MCQs

Standardized Patients

Faculty Observation

Impact on Patient

Clinical vignettes

Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9 Suppl), S63-7.


WBA

•  Kogan and Holmboe (2013) define WBA:

•  “WBA is the assessment of trainees and physicians across the continuum of day to day competencies and practice in authentic, clinical environments…..it enables the evaluation of performance in context” (p. S68).

55

Mini-MAS/Ladhani


WBA

•  The In-Training Evaluation Report (ITER): »  Does not discriminate (Gray, 1996; Holmboe & Hawkins, 1998 )

»  Completed retrospectively (Turnbull et al., 1998).

»  Often faculty who have not observed are completing the form (Epstein, 2007)

»  Halo effect (Wilkinson & Wade, 2007)

»  Raters fail to use the entire scale (Gray, 1996)

56

Mini-MAS/Ladhani


Mini-CEX

•  The scale used in the mini-CEX is designed for linear gradations of performance. The scores do not give the evaluators a point of reference to help align a trainee to a category or score (Crossley & Jolly, 2012).

•  Faculty assessors resort to norm-referencing.

57

Mini-MAS/Ladhani


Mini-CEX

•  Hawkins et al. (2010) The raters did not use the full nine-point scale, and the distribution is right shifted towards the higher end of the scale, the use of the lower end of the scale was infrequent raising concerns about identifying weaknesses.

•  Individual competencies tended to be highly correlated. This phenomenon is likely related to the fact the rating form had overlapping descriptors

58

Mini-MAS/Ladhani


Designing a Better Tool

•  WBA assessment tools should have anchors linked to the construct of clinical independence, measuring the trainees’ level of progression and development (Crossley et al., 2011).

•  Assessors make more reliable judgments of performances

they can see clearly in a particular context or activity. •  The tool should focus on the competence relevant to the

activity, and avoid having multiple competencies to assess at the same time (Crossley & Jolly, 2012).

59

Mini-MAS/Ladhani

Figure 2 General Curve of skills Acquisition. (ten Cate et al., 2010)

The Campaign for McMaster University The Campaign for McMaster University


61

Mini-MAS/Ladhani 2 a week, one competency at a time, 6 competencies 40-44


6 Competencies

The assessment of competencies was limited to six core competencies:

»  history taking, »  physical exam, »  clinical reasoning, »  communication with families, »  communication with physicians and other health care

professionals and »  collaboration

62

Mini-MAS/Ladhani


•  A total of 21 residents were included in the study.

•  12 PGY 1 residents at McMaster Children’s Hospital were required to have completed 40 unique observations in six domains over 22 weeks.

•  In addition, nine PGY 4 residents were also required to be observed for the same competencies over the same time period. This group was required to complete 15-20 encounters as their general pediatric exposure was less during this final year of training.

•  Faculty and Senior residents assessors were trained-Process

63

Mini-MAS/Ladhani


Results

64

Mini-MAS/Ladhani

The International Conference on Residency Education | La Conférence internationale sur la formation des résidents 65

Mini-MAS/Ladhani


Mini-MAS/Ladhani

The Campaign for McMaster University The Campaign for McMaster University


Mini-MAS/Ladhani


74

Mini-MAS/Ladhani


Resident Comments

•  “ I do believe this will influence my learning positively”,

•  “the Mini-MAS book is a good tool for assessing the learners in their visual environment” and

•  “The Mini-MAS is arguably a good tracking tool…given the culture of medicine and the temptation for trainees to avoid supervision”

75


Conclusion

•  Work based assessment tool should achieve three requirements; »  the competences expected as outcomes and the

assessment should be aligned; »  feedback is provided during and or after the assessment »  the assessment is used to guide a trainee towards a

desired outcome (Norcini & Burch, 2007).

•  The Mini-MAS tool has achieved these three requirements and has shifted the focus of assessment from assessment of the trainee to learning of the trainee. (Driessen & Scheele, 2013)

76

Mini-MAS/Ladhani


Conclusion

•  The Mini-MAS added as a formative assessment mode to a multi-modal assessment program will benefit the trainee, informing them on where they stand compared to their level of training, what competencies they can improve on and how they can do that.

77

Mini-MAS/Ladhani

Features of Successful Change

•  Educational institutions with a history of effective change are more likely to implement new innovations successfully.

•  Successful change efforts are characterized by:

»  (a) having a strong, influential advocate at the forefront of the change effort.

•  Organizations with high interaction, connection, and networks of participatory teams are better able to accomplish broad change than ‘‘segmental,’’ ‘‘departmentalized,’’ or ‘‘loosely coupled’’ organizations

•  There needs to be widespread agreement that the

innovation is relevant to the real problems that potential users currently experience. 80


•  Avoid overly ambitious undertakings. A balanced approach that engages the organization members’ commitment and support will be most successful.

•  The importance of a positive, respectful work climate to successful curricular change cannot be overstated.

•  Successful innovation is fostered by frequent, timely, substantive, and forthright communication.

•  Faculty development and training is vital

81


•  Formative evaluation is useful in locating difficulties and solving problems .

•  Stable leadership is positively associated with successful innovation.

•  Bland et al, Curricular Change in Medical Schools: How to Succeed, ACADEMIC MEDICINE, VOL. 75, NO .6 / JUNE 2000

82

Thank You

83