Lee Atkinson-McEvoy, MDGlenn Rosenbluth, MDUCSF Pediatric Residency Training Program

Planning to implement in your program? Learners (students , residents or fellows)

have used it at other institutions and are asking for your participation?

Future of medical education, want to start within your program?

Introduction Portfolio vocabulary Examples of IDPs in electronic portfolios How to create a IDP in an electronic

portfolio platform Are they worth the effort?

Introduction What is the purpose of this workshop? Why are we talking about portfolios?

(Why is this culture emerging?)

Understand the purpose of portfolios and how they can be used to create Independent Development Plans for learners.

Introduction to the process of setting up a portfolio for learners

Create a work book to use to implement an IDP within your program.

Introduction Portfolio vocabulary

“A portfolio is a purposeful and longitudinal collection of tangible evidence of learner-selected work that exhibits the learner’s efforts, progress or achievement.

This portfolio features the criteria for selection and judging merit, and includes evidence of learner reflection.”

UCSF EPIC Report, Adapted from the Committee on Student Assessment’s portfolio definition from Reckase, 1995.

Formative or Learning Portfolio: Portfolio used to aid learning, with accompanying feedback to improve performance

Summative or Assessment Portfolio:Portfolio used for evaluating and / or grading a learner’s performance for the purpose of determining their readiness for advancing to the next phase of an educational program

Work products that learners can use to document competency

Should be embedded within their work. Should be part of curriculum Examples: Test results Poster/abstract/journal article Letter from a patient or a patient's family member Resume/CV Select narrative assessments from E*Value Slides from a lecture Blog entry or essay about experience with patient

Critical reflection: is analysis of personal experience to enhance learning and improve future behavior and outcomes. Without it, portfolio would not be a portfolio but

a database. By critically reflecting on the artifacts in their

portfolios, learners identify their own strengths, opportunities for improvement, and development over time.

Goes beyond “consideration or meditation” A skill developed over time for learning and

lifelong professional development. Not just what happened (anecdote) More than your personal opinion of events Requires Data gathering Integration of past, present and future

experience Analysis and reframing

Adapted from Louise Aronson’s presentation on Critical Reflection

Critical reflection should describe how a piece of evidence is related to the

educational competency the learner is addressing what was learned about the competency what the learner discovered about themselves

(as a learner, clinician or human being) from the experience how this discovery can be used to further

professional development

Presentation Portfolio (or view): Display of evidence chosen by learner with reflection, organized and communicated to an advisor, mentor or others for a specific purpose

Learning Plan: is a learner-specific program or strategy for

learning Integration of critical reflections based upon

evidence for all areas into a plan which informs an efficient process for learning

Review of the learning plan by an advisor, mentor or program/course director.

Should have specific criteria, guidelines by which to assess where the learner is within the program.

If part of a summative evaluation some formal rubric should be used.

This information should be available to the learner to assist in the critical reflection and development of a learning plan.

Introduction Portfolio vocabulary Examples of Individual Development Plans

in electronic portfolios

Introduction Portfolio vocabulary Examples of Individual Development Plans

in electronic portfolios How to create a Individual Development

Plan in an electronic portfolio platform

Portfolio Step* Responsible party

1. Collect Evidence Program/trainee

2. Reflection Trainee

3. Evaluate evidence Mentor/Trainee

4. Defense Trainee to program

5. Summative Decision Program

* Friedman Ben David, M, Davis, MH, Harden, RM, et al. (2001). AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical teacher, 23(6), 535-551.

What are residents to document in their portfolio?

Why document this way versus another way?

How will it be used? Formative vs. summative feedback?

Who will have access to it?

ACGME Competencies Program Specific Professional

Competencies Career Development Personal Development Learning Plans

Learners need a guide for what is expected with each entry.

How to pick evidence—criteria for selection What supporting information can be used

as evidence.

Provides context for artifact. Guides mentor/advisor in giving feedback Allows for evaluation on summative

feedback for assessment of competency.

Necessary for standardization of assessments.

Helps guide the learner in identifying where they should be for their level of professional development.

Rubric useful tool to standardize assessment

Rubrics: multidimensional scoring guidelines used to provide consistency in evaluating learner work elucidate scoring criteria so multiple evaluators,

using the same rubric, would arrive at same score or grade

provide a measurement system for specific tasks and are tailored to each project

let learners know what is expected of them provide learners the opportunity to do self-

assessment to reflect on the learning process

Does not meet Needs improvement Meets Exceeds

Selects evidence that demonstrates current progress relative to the milestone

Evidence does not provide information on the skills/ activities listed in the milestone

Appropriate evidence selected, but provides limited details to be able to infer progress toward the milestone; no information in reflection that provides context for the evidence

Evidence demonstrates the learner can perform at the level of the milestone or has a plan that makes achieving the milestone likely

Evidence supports achievement of the milestone and has characteristics of year two milestone

Provides a thoughtful reflection and assessment of the learner’s progress toward competency in the domain

Describes a situation that does not address domain skills; perhaps focusing on someone else’s situation; does not take the reflection seriously

Discusses a specific experience, but does not indicate critical reflection; no lessons learned are drawn from the evidence/experience

Learner described the value of the experience/evidence and articulated a fundamental lesson learned

Learner described the experience and addressed multiple and/or complex lessons learned.

Describes how feedback has informed learning in the domain.

Does not indicate what feedback was obtained to help develop competency toward milestone.

Description relies solely on personal perception about what and/or how to improve in this domain.

Based on own perceptions and one other source formulates a plan that is consistent with evidence provided for this domain.

Indicates more than one source of feedback and clearly describes how that information has guided learning.

Describes strategies for future growth in the domain

Not clear what the learner’s strategies are for future growth

Describes strategies for future growth but does not articulate specific resources or plans

Clearly identifies relevant next steps for growth in the domain including resources, strategies that are timely and feasible

Describes specific, timely, and feasible plan for future growth in the domain and considers personal and professional challenges, and strengths

Benchmarks: A reference point that serves as a basis for judging whether a student has met the associated educational standard.

Hardware—computers Software—electronic portfolio platform Time—orientation for residents and faculty,

time to complete portfolio, time to review/evaluate entries

Manpower—faculty to review, administrative staff to collect data, reminders etc.

Faculty Should know what is expected of the learner Give feedback in a timely fashion Use a rubric if provided to guide feedback

Peer(s) Can be used to provide feedback Must follow the same guidelines as the faculty

Portfolio Step* Responsible party

1. Collect Evidence Program/trainee

2. Reflection Trainee

3. Evaluate evidence Mentor/Trainee

4. Defense Trainee to program

5. Summative Decision Program

* Friedman Ben David, M, Davis, MH, Harden, RM, et al. (2001). AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical teacher, 23(6), 535-551.

Clear Purpose Make sure you have your resources available Provide criteria for selection/specific

instructions to guide selection of evidence Require reflection Set clear expectations As to how much evidence (remember lean is good) Clear timeline Clearly communicate to other faculty their role Make sure that feedback is timely

What incentives are there for the learners to complete this portfolio? (there have to be teeth in the process for most learners)

What barriers do you anticipate for: Learners Faculty

Suggestions to overcome barriers…

Introduction Portfolio vocabulary Examples of Individual Development Plans

in electronic portfolios How to create a Individual Development

Plan in an electronic portfolio platform Are they worth the effort?

Are learning portfolios worth the effort? No Its versatility makes it hard to judge whether a portfolio is

good or bad Limited evidence; much obtained from self-report Not clear that effort is rewarded Benefit may be in a side-effect: forces students to write something and teachers to spend some

time with students

Norman, G. BMJ 2008:3337:a514

Are learning portfolios worth the effort? Yes

Portfolio is best solution since it allows the collation and integration of evidence

Portfolios can guide and coach professional development

Flexibility is advantage when carefully developed Mentoring is the single most decisive success

factors Portfolios must be smart and lean Careful implementation is crucialDriessen, E. BMJ 2008:337:a513

Office of Medical Education UCSF School of Medicine

Daniel C. West, MD Patricia O’Sullivan, EdD Lorriana Leard, MD Louise Aronson, MD Chandler Mayfield

Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, Peackock H. The Effectiveness of Portfolios for Post-Graduate Assessment and Education: BEME Guide No. 12. Med Teach. 2009;31:299-318.

Colbert CY, Ownby AR, Butler PM. A review of portfolio use in residency programs and considerations before implementation. Teach Learn Med. 2008 Oct-Dec;20(4):340-5.

Carraccio C, Englander R. Evaluating competence using a portfolio: a literature review and web-based application to the ACGME competencies. Teach Learn Med. 2004 Fall;16(4):381-7.

Norman G. and Driessen E. Are Portfolios Worth the Effort. BMJ 2008; 337:a514