Portfolio assessment of behavioral competencies at the Feinberg School of Medicine Celia Laird O’Brien, PhD

Competency Based Medical Education

2

3

Competency Based Medical Education

Competency Based Medical Education

4

Does

Shows How

Knows How

Knows MCQ exam

Clinical vignette, essays

OSCE, simulation

Workplace Based Assessment

• One observation isn’t enough.

5

Assessment of behaviors

Assessment of behaviors • Many data points are necessary, collected from

multiple observers, in different contexts, across time.

6

Assessment of behaviors

• How do we organize and review this data to determine competence?

7

Result: Lots of data!

The portfolio approach

8

• The Feinberg electronic portfolio is used to organize this data: a “learning chart”

• Allows students to review and reflect on their performance across time

• Allows us to measure and enhance behavioral competencies in our students

The portfolio approach

Three goals:

1. Permit assessment of competencies not easily measured by traditional grading methods. 2. Develop students’ capacity to self-assess their skills and abilities using external feedback. 3. Develop students’ ability to engage in self-directed learning and improvement.

9

Assessment at Feinberg

10

Fail/Pass/High Pass/Honors Pass/Fail Fail/Pass/Honors

Traditional grading system

Portfolio review Portfolio review

The portfolio approach

11

Competencies subject to portfolio review:

• Effective Communication and Interpersonal Skills (ECIS)

• Patient Centered Medical Care (PCMC)

• Professional Behavior and Moral Reasoning (PBMR)

• Systems Awareness and Team Based Care (SATBC)

• Continuous Learning and Quality Improvement (CLQI)

The portfolio approach Examples of assessment data:

• Faculty and peer evaluations of small group work

• Clinical performance evaluations

• Feedback from standardized patients

• Feedback from nurses

•Narrative data is most informative. 12

13

14

15

16

Portfolio review process

• Portfolio review committees comprised of experienced clinician educators

• Each portfolio is reviewed by at least two clinicians, who judge competence by the following:

a) Contents of the portfolio (institutional assessment data, external uploads)

b) Quality of reflections and learning plans

17

Portfolio review process •Decisions:

• Progressing towards competence

• Progressing towards competence with some concern

• Progressing towards competence pending additional development

18

Portfolio review process •Assessment of competence isn’t about the numbers.

•Process depends on judgment from experienced educators.

• Involves analysis of qualitative data; uses qualitative methodology to collect evidence of validity

19

Portfolio review process •Reviewers work in pairs

• Independently score portfolios, then come together to reach consensus through discussion and debate

•Third reviewer used if consensus cannot be reached

•Final decisions reviewed by committee as a whole

20

Portfolio review process •Multiple training sessions are held, including group standard setting exercise

• Important to work from a shared frame of reference

•Reviewers look for patterns and themes in performance across time

21

Results – Phase 1

22

25

39

30 38

16

1

4

4

4

1

0

5

10

15

20

25

30

35

40

45

50

CLQI ECIS PCMC PBMR SATBC

PC PAD

Evaluation

• Statistical analyses show a significant association between results of the portfolio review in Phase 1 and clerkship performance in Phase 2, even after controlling for medical knowledge.

23

Evaluation

24

3

2

1 1

2

3

0

1

2

3

4

5

6

7

Honors High Pass Pass

Median number of clerkship grades awarded by group

No concerning behavior (n=111) Concerning behavior (n=24)

p<.01 p<.01

Evaluation

25

82.3

79.4

70

72

74

76

78

80

82

84

86

88

90

No concerning behavior Concerning behavior

Mean clerkship performance score

p<.001

26

Lessons learned

27

“The last two decades have been characterized by happy

portfolio developers and grumpy portfolio users.”

- Erik Driessen, 2016

Lessons learned

• Many students don’t see value of mandated reflection or self-improvement efforts

28

29

4.4

3.2

1

2

3

4

5

My mentor is able to helpme understand my

evaluation data as it relatesto the competency

standards.

Writing reflections about myprogress in the

competencies has proved tobe a useful exercise.

Strongly agree

Agree

Neutral

Disagree

Strongly disagree

Lessons learned

• Numbered ratings proved to be misleading and often contradicted narrative comments.

• Vast majority of students are rated “above average”… even when they’re not.

30

Lessons learned

• There is inherent overlap between these competency domains.

31

Communication Professionalism Teamwork

Lessons learned

• We need to focus more on training our reviewers to give effective feedback.

32

Lessons learned

• We need to re-examine our remediation efforts and determine how to best support students with deficiencies.

33

Lessons learned

• We had to pull back plans to transmit results of the portfolio review to residency directors

34

Conclusion

• Portfolio assessment is a feasible way to assess behavioral competencies but requires investment of dedicated time and resources.

35

References 1. O’Brien C.L., Sanguino S.M., Thomas J.X., Green M.M. Feasibility and outcomes of implementing a portfolio

assessment system alongside a traditional grading system [published online ahead of print March 29, 2016]. Academic Medicine.

2. Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Academic Medicine. 2007;82(5):493-502.

3. Driessen E, van der Vleuten C, Schuwirth L, van Tartwijk J, Vermunt J. The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study. Medical Education. 2005;39(2):214-220.

4. Johnston B. Summative Assessment of Portfolios: An Examination of Different Approaches to Agreement over Outcomes. Studies in Higher Education. 2004;29(3):395-412.

5. Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Academic Medicine. 2010;85(7):1212-1220.

6. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Medical Teacher. 2012;34(3):205-214.

36

QUESTIONS?

37