sources of outcome data internal measures –end of course & clerkship surveys –end of year...

SOURCES OF OUTCOME DATA

• Internal measures– End of course &

clerkship surveys– End of year surveys– Faculty survey– Universal Student

Rating of Instruction (USRI)

– Certifying exam scores– Canadian Graduate

Questionnaire

• External measures– MCC– Resident program directors’

evaluation of graduates – NBME-Comprehensive

Basic Science Exam– CaRMs– Alumni– LCME report– Alberta

Universities/Colleges Graduate Employment Survey

A Jones, Associate Dean – University of Calgary

Overall Rating of First Year Courses

1

1.5

2

2.5

3

3.5

4

4.5

5

POM Blood MSK CV Resp Renal Endo Integr RMEBM

2000 2001 2002 2003 2004 2005 2006 2007

Excellent

Poor

Good

V.Good

Fair

Mean Scores on Certifying Evaluations (System Courses-Yr 1; SB vs CP)

60

65

70

75

80

85

POM Blood MSK Card Resp Renal Endo

Mea

n Sc

ore

SB (93-96)CP (97-07)


Mean Scores on Certifying Evaluations (System Courses-Yr 2; SB vs. CP)

60

65

70

75

80

85

Neuro Mind Repro GI

Mea

n Sc

ore

SB (93-96)CP (97-06)


ALBERTA LEARNING GRADUATE EMPLOYMENTSURVEY 2004 GRADUATES FROM 2002 MEDICINE

Usefulness of Your Education in Achieving:

• Research Skills 80%

• Working with Others 97%

• A Desire to Learn More 93%

• Learn Independently 97%

• Awareness of Ethical Issues 97%


ALBERTA LEARNING GRADUATE EMPLOYMENT SURVEY 2004

GRADUATES FROM 2002MEDICINE

• Satisfaction with the quality of teaching in your program? 100%

• Satisfaction with overall qualityof your educational experience 100%

• University of Alberta 83%


ALBERTA LEARNING GRADUATE EMPLOYMENT SURVEY 2004

GRADUATES FROM 2002MEDICINE

• I would recommend the same program of study to

someone else. 100%

• Satisfaction with Relevance of Courses 96%


Overall Quality of Education at U of C by Faculty: % Satisfied or Very Satisfied

0

10

20

30

40

50

60

70

80

90

100

FA HU SS CC SC HA ED KN EN NU SW LA MD

Data Source: 2002 Alberta Universities/Colleges’ Graduate Employment Survey re: 2000 Grads

PREPAREDNESS FOR RESIDENCY

1

1.5

2

2.5

3

3.5

4

4.5

5

1 2 3 4 5 6

U of C 05

All Schools 05No Opinion

Strongly Disagree

Data Source: Canadian Graduate Questionnaire 2005

1: I am confident that I have acquired the clinical skills required to begin a residency program

2. I have the communication skills necessary to interact with patients and health professionals

3. I have basic skills in clinical decision making and the application of evidence based information to medical practice

4. I have the fundamental understanding of the issues in social sciences of medicine

5. I have the ethical and professional values that are expected of the profession

6. I have the fundamental understanding of the basic disease mechanisms, clinical presentations and principles of diagnosis and management for common conditions

Strongly Agree

Agree

Disagree

“I AM SATISFIED WITH THE QUALITY OF MY MEDICAL EDUCATION”

0

10

20

30

40

50

60

70

Strongly Agree Agree No Opinion Disagree StronglyDisagree

Per

cent

U of C 03

All Achools 03

U of C 04

All Schools 04

U of C 05

All Schools 05

Data Source: Canadian Graduate Questionnaire 2003, 2004 & 2005A Jones, Associate Dean – University of Calgary

MCC TOTAL SCORE: 1992-2005

440

460

480

500

520

540

560

580

92 93 94 95 96 97 98 99 0 1 2 3 4 5

Class

Mea

n To

tal S

core

U of C

Canadian


Performance on national exams

Performance on LMCC – Clinical Reasoning: 1994-2002

470

480

490

500

510

520

530

540

550

560

94 95 96 97 98 99 0 1 2

Class

Mean

Sco

re

U of C

Canadian


720 – U of C888 - Canadian Grads/Canadian Trained


328 – U of C

111 - Canadian Grads/Canadian Trained

MCC Subscale & Final Scores– Class 2003

Canadian (84), International Students (7), Total (91)

360

380

400

420

440

460

480

500

520

540

560

C leo Med O & G P eds P hello P sy S urg MC Q C R S Final

Mea

n S

core U of C Ca n

Inte rna tiona l

U of C Tota l

M CC M e a n


CaRMS: PERCENT OF MATCHED STUDENTS MATCHING TO FIRST CHOICE DISCIPLINE IN 1ST ITERATION

CLASSES 2001- 2005

50

55

60

65

70

75

80

85

90

95

100

Per

cen

t

Mem Dal McG Ott Qu TO McM West Man Sk AB Cal BC

1 2 3 4 5

Data Source: CaRMSA Jones, Associate Dean – University of Calgary

RESIDENT DIRECTORS’ ASSESSMENT OF GRADUATES (PGY1)

“Overall Performance - ability to function as a resident with a full workload”

6

32

62

4

32

64

44

56

1

37

62

40

60

0

10

20

30

40

50

60

70

80

90

100

Per

cen

t

2000 2001 2002 2003 2004

Weaker than most Similar to most residents Stronger than most

Data Source: Program Directors’ Survey

Class 2000 N = 50 (71%); Class 2001 N = 45 (68%) Class 2002 N = 40 (57%); Class 2003 N = 79(90%)Class 2004 N = 76 (82%)


Resident Program Directors’ Assessmentof 2006 Graduates

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Medic

al Knowle

dge

Clinic

al Ju

dgmen

t

Patient M

anagem

ent

Clinic

al Ski

lls

Profe

ssion

al D

emea

nour

Human

istic

Qual

ities

Presen

tatio

n Ski

lls

Self-Aware

ness

Psycho

social S

ensi

tivity

Perform

ance O

vera

ll

Per

cen

t

Weaker Similar Stronger

Data Source: Program Directors Survey



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Med

ical

Knowle

dge

Clinica

l Ju

dgmen

t

Patie

nt Man

agem

ent

Clinica

l Ski

lls

Profe

ssional D

emea

nour

Human

istic

Qual

ities

Prese

ntat

ion S

kills

Self-A

waren

ess

Psych

osocial S

ensi

tivity

Perfo

rman

ce O

veral

l

Per

cen

t




UNDERGRADUATE MEDICAL EDUCATION ALUMNI SURVEY

CLASSES of 1992-2002

A Jones, Associate Dean

OVERALL OPINION OF THE UNDERGRADUATE MEDICAL

EDUCATION PROGRAM ALUMNI SURVEY CLASSES 1992-2002

• 97% Satisfaction with the UME program at University of Calgary

• 90% Felt prepared or very prepared for Post Graduate Training

• 98% Would advise their child or child of a relative or friend interested in Medicine to apply to the University of Calgary


OPNION OF ALUMNI 1992 – 2002CURRICULUM STRENGTHS AND WEAKNESSES

Instruction Strength Neither Strength or Weakness

Bedside Clinical Correlation 92% 8%

Clinical Instruction Overall

88% 11%

Small Group Problem Solving

86% 12%

Classroom Patient Presentation

83% 15%

Learning Experiences Appropriate to Educational Objectives

80% 18%

Lectures 63% 31%

Integration of Basic and Clinical Sciences

53% 33%

“A curriculum is like water. It has the tendency to seek the lowest level of energy it can reach, and without constant renewal, it will stagnate and become putrid. To avoid stagnation alone is justification for action.”

Acad Medicine Sept 1998

Why Curriculum renewal is Important


Barriers to Medical School Curriculum Changes

• Already crowded curriculum• Inadequate funding• Faculty resistance• Professional ‘turf’ issues• Scheduling conflicts

Listed by North American Academic Deans:

Graber et al. Acad Medicine 1997


Curriculum – A Planned Educational Experience

• Define the outcome measures.

• Create an evaluation system to be sure these outcomes are realized.

• Develop the pathways to get to these outcomes.

Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education

Goals for a Revised Curriculum

A revised curriculum has to be consistent with available information on clinical problem solving and reflect basic principles of adult learning.


Medical students don’t remember or can’t use the knowledge they learned in the traditional basic science courses because the knowledge is structured into mental organizations that are not useful in the clinic

Barrows, 1985


Problem Based Learning – Benefits

• Activate prior knowledge

• Learn in context of clinical problem

• Interest in learning stimulated

• Self directed learning encouraged

• Life long learning encouraged

Schmidt - Norman


Problem Based Learning Concerns

• Problem solving skills are not augmented

• Significant gaps in knowledge occur

• Incorrect integration of basic sciences

• Tendency to engage in backward reasoning

Albanese; MitchellAcademic Medicine


Clinical Reasoning

• Clinical Reasoning and clinical knowledge are interdependent.

• Effective problem solving requires a large store of relevant knowledge.

• Clinical expertise is linked to depth and organization of clinical knowledge.


Problem Solving Skills In Medicine

Research has proven that experts in specific domains learn knowledge and problem solving skills for each problem simultaneously. That is, knowledge acquisition and clinical reasoning go hand-in-hand.

Schmidt et al 1992

Efforts to Help Students Improve Clinical Reasoning

Education must focus on the development of adequate knowledge structures. Teaching, coaching, supervising must strongly encourage and nurture actual knowledge organization of the students.

Knowledge keeps no better than fish

Alfred North Whitehead 1929

Clinical Reasoning and Small Group Cases

It is useful to select one model of clinical reasoning and base the tutorial discussion on it. The precise model is less important than its generic use as a framework to structure the flow of discussion. It later serves as a fall-back strategy in complicated clinical situations.

Structure of Medical Knowledge in Memory Categories and Prototypes

Both medical textbooks and classroom teaching abound in the limitless presentation of detailed lists of disorders. More often, both fail to provide a categorization scheme that is best suited for their retrieval in a clinical problem solving situation.

BordageMed Educ 1984

Types of Curricula

• Disciplinary

• Systems-based

• Problem-oriented

• Clinical Presentations based

Clinical Presentation Curriculum

Faculty

Identify Represented byIdentify

Clinical Presentation

Schematic Problem Solving Pathway

Develop

For the Process of

Core Competencies for Clinical Presentation

Clinical Reasoning

Enabling Basic Science

Objectives

Terminal Objectives

Together Represent

Graduation Competencies

Curriculum Committee

Plans and Monitors

Curriculum

Course Content

Teaching Methods

Learning Content

Evaluation

Guidelines for

Steps in Development and Dissemination of Clinical Presentation Objectives

1. Selection of clinical problem.

2. Classification system developed to help organize knowledge needed to solve the clinical problem.

3. Key Features; Discriminating features identified of prototypic prevalent disorders.


4. Objectives and problem solving schemes developed.

5. Distribution to Faculty for balanced input from teachers generalists, specialists, and biomedical scientists.

6. Endorsements of objectives.


7. Dissemination of objectives.

8. Encouragement of implementation of objectives in teaching, learning, clinical practice and problem solving

9. Monitor and evaluate the translation of objectives and problem solving schemes into practice.

Clinical Reasoning

Student Identifies Clinical Presentation

Broad Classification of Problem

Identify Causal Alternatives and Discriminating Key Factors

Differential Diagnosis

Diagnosis

Management Plan


The Scheme

• Causal Categories- pre, post and renal causes of acute renal failure

• Diagnoses- specific diagnoses for each causal category

• Basic sciences- Integral part

- Timely presentation of content

“Ask any physician of 20 years standing how he has become proficient in his art and he will reply, by constant contact with disease; and he will add that the medicine he learned in schools was totally different from the medicine he learned at the bedside.”

Wm. Osler 1932

Bleeding Tendency/Bruising

Hx PE DDx Invest. NatHx Mgmt

General Objectives

Thrombocytopenia

DisorderedPlatelet Function

CongenitalCoagulation Disorders

Acquired CoagulationDisorders

VascularAbnormalities

W SurgeryV Student

U RadiologyT Psychiatry

S PhysiologyR Pharmacology

Q PediatricsO Pathology

N OncologyM Office of Medical Education/Informatics/Culture, Health and Illness

L Office of Medical BioethicsK Obstetrics & Gynecology

J NeuroscienceI Microbiology

H MedicineG Immunology

F GeneticsE Family Medicine

D Community Health Sciences/Nutrition/PreventionC Biochemistry

B AnesthesiaA Anatomy

Bleeding Tendency/ Bruising

History Physical Examination


Investigation

Natural History, Prognosis &

Complications of Condition

Prevention, Treatment &

Complications of Treatment

GeneralObjectives

Thrombocytopenia

Disordered Platelet Function

Congenital Coagulation Disorders

Acquired Coagulation Disorders

Vascular Abnormalities

a

b

c

d

e

f

g

0 1 2 3 4 5 6



Platelets Coagulation Vascular

Decreased Number

Abnormal Function

Congenital Acquired Congenital Acquired

Clinical Presentations

Broad Classification of Problem

Causal Alternatives and Discriminating

Key Factors


Basic Science Objectives for Bruising and

Bleeding

Pathology (Vessels)

Anatomy (Spleen)

Histology (Bone Marrow)

Pharmacology (ASA, Heparin)

Immunology (ITP, Vasculitis)

Genetics (Hemophilia)

Physiology (Hemophilia)

Basic Science or Biomedical Knowledge in the Undergraduate

Program The purpose of basic science teaching is to

provide a scientific foundation for tasks of clinical practice such as diagnosis and therapeutics. The essential challenge of balancing depth of understanding with breadth of coverage remains.

(See p. 35, Fig. 4.1)

W SurgeryV Student

U RadiologyT Psychiatry

S PhysiologyR Pharmacology

Q PediatricsO Pathology

N OncologyM Office of Medical Education/Informatics/Culture, Health and Illness

L Office of Medical BioethicsK Obstetrics & Gynecology

J NeuroscienceI Microbiology

H MedicineG Immunology

F GeneticsE Family Medicine

D Community Health Sciences/Nutrition/PreventionC Biochemistry

B AnesthesiaA Anatomy


History Physical Examination


Investigation

Natural History, Prognosis &

Complications of Condition

Prevention, Treatment &

Complications of Treatment

GeneralObjectives

Thrombocytopenia

Disordered Platelet Function

Congenital Coagulation Disorders

Acquired Coagulation Disorders

Vascular Abnormalities

a

b

c

d

e

f

g

0 1 2 3 4 5 6

Fever

Fever

< 2 weeks > 2 weeks

Infectious

Bacterial Viral

ACUTE VISUAL LOSS

PreRetinal• Corneal edema (glaucoma)• Vitreous hemorrhage (Diabetes)

Retinal• Acute Macular Lesion (hemorrhage)• Retinal Detachment (spontaneous)• Retinal Artery Occlusion (carotid emboli)

Post Retinal • Optic Neuritis (MS)• Ischemic Optic Neuropathy (Temp. Arteritis)• Occipital Infarction/Hemorrhage

Chrichton, Verstraton, Fletcher

Fluctuating Altered CognitionFluctuating Altered Cognition Delirium and Confusional States

In the Head Out of the Head

•CNS Infections•Seizures

•Hypertensive Enceph•Psychiatric Disorders

•Toxins•Metabolic Derangements•Systemic Organ Failure

•Physical Disorders

Strengths of Year I-II Curriculum

• Approaches to problem solving – clinical reasoning

• Early clinical exposure• Small group teaching• Clinical correlation; patient presentations• Basic science integration with problem

solving• IST • Medicine 440; elective time• Communication; physical examinationAllan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education

Weakness of Year I-II Curriculum

• Exams not always reflective of teaching ‘emphasis’

• Lack of pharmacology• Faculty not promoting core documents;

teaching to objectives not always clear• Small group teaching variable• Problem solving with schemes course

dependent



The Curriculum First Year Courses

Aug Oct Nov Dec 19 Jan.2 Mar. 26 Mar. 29 June -July 18

P FOR M-Blood-GI(Course I)

MSK-Derm-Opth-ENT

(Course II)

HOLIDAY

2 wks

CV-RESP

(Course III)

Renal-Endocrine(includes

Obesity P for M)

(Course IV)

HOLIDAY

2wks

ELECTIVE

2wks

REWRITES

MEDICAL SKILLS PROGRAM

RMEBM-Health Promotions-Disease Prevention-Population Health

Number of weeks for First Year = 45 weeks


The Curriculum

Second Year Courses

July 18

Aug. 18 Oct. 10 XMAS Jan. 3Feb. 6 Mar. 5

ELECTIVE

4wks

Neuroscience-Aging

(Course V)

Infant-Child-Reproduction-

Genetics(Course VI)

HOLIDAY

2 wks

Mind-Family(Course

VII)

IntegrativeCourse

Introduction to Clerkship

Teaching Methods

• Lectures for rapid acquisition of key content

• Small group case based learning- In depth self-directed (or guided) learning

- Review, reinforcement, practice and feedback

- Problem solving, motivation, pertinence

• Clinical correlation- Bedside sessions

Clinical Reasoning

Clinical reasoning does not develop in isolation: it is associated with increasingly refined and elaborated medical knowledge. Problem solving is domain-specific and not generic, so the challenge for medical educators is not only to make explicit the process of reasoning but also to identify the necessary content.

Schmidt et al 1990

Kassirer 1995

Usual Sequence of Instruction

• Presentation and scheme shown- Case based or otherwise

• Lectures or PBL sessions planned- In depth knowledge and acquisition

- Basic and clinical sciences

• Small group sessions for reinforcement- Shorter case scenarios for review

- Clinical correlation

Advantages of New Curricular Structure

• Courses will be linked to graduation objectives and UME program philosophy of teaching, learning and evaluation.

• Linkage of courses will integrate CP better and reduce redundancies.

Curriculum

Task Force Report

Advantages of New Curricular Structure

• Clinical presentation list will be revisited and clerkships will adopt appropriate presentations.

Curriculum

Task Force Report

Scheme Use Reported by First & Second Year Students(Classes of 2007 and 2006)

1

1.5

2

2.5

3

3.5

4

4.5

5

Learning Prob-Solving Often Referred Integration

1st Yr 2nd Yr

Strongly Agree

Strongly Disagree

Neutral

Data Source: Classes 07 & 06 yr end CP curriculum evaluation

MAP OF DEPARTMENT INVOLVEMENT IN UME CURRICULUM*COURSES IN YEARS I AND II

Department I II III IV HPOP V VI VII Int. Intro Clerk

MedSkills

Medicine X X X X X X X X X

Family Med X X X X X

Pediatrics X X X X ? X

Psychiatry X X X X

Surgery X X X X X ?

Neurosci X X

Anatomy X X X X X

Oncology X X X X

Obst-Gyn X

Community Health

X

Medical Genetics

X

Pathology X X X

ER X X X

Anesthesia X

NUMBER OF HOURS TAUGHT IN THE UNDERGRADUATE MEDICAL PROGRAM YEARS I-II

GFT and CLINICAL FACULTY

Hours Relative Dept Size Average Teaching hrs per Member

Medicine 1844 7

Family Med 1189 7

Pediatrics 694 5

Neuroscience 514 8

Psychiatry 458 4

Surgery 437 3

Pathology 257 4

Anatomy 252 10*

Oncology 242 3

Obst-Gyn 155 6

Com Health 78 1

Med Genetics 51 4

LEARNING METHODS USED DURING YEARS I AND II UNIVERSITY OF CALGARY

UNDERGRADUATE PROGRAM

Small Groups/Bedside: 1/3

Large Groups/Classroom: 1/3

Self Directed Study: 1/3


SUMMARY OF STRUCTURED TEACHING HOURSUNDERGRADUATE MEDICAL EDUCATION PROGRAM

YEARS I AND II 2004-2005

Hours %

Non-GFT 5316 68

GFT 2453 32

TOTAL 7769 100


*Medicine 440, Summer Elective not included

Ownership of the Curriculum

"Faculty members own what is taught in the curriculum - they own the content. The Associate Dean and Curriculum Committee are responsible for the methods and the effectiveness. Faculty give the course, the Associate Dean has to give the degree.”

Academic MedicineVol. 73, Pg. 54

The Seven Most Dangerous Words in Medical Education

“…but we’ve always done it this way”


1. Decide what to do.

2. Build support for the idea.

3. Acquire resources.

4. Do it wrong the first time.

5. Do it wrong the second time.

6. Do it passably the third time.

7. Do it reasonably well the fourth time.

Friedman Acad Medicine 1993

Prototype Timeline for a Major Educational Change

sources of outcome data internal measures –end of course & clerkship surveys –end of year...

Documents

university of alberta

good fair slide

satisfied data source

medical education data

medicine satisfaction

overall quality of education

u of c

basic skills