ags annual meeting may 2005 converting the geriatric functional assessment standardized patient...

AGS Annual MeetingMay 2005

Converting the Geriatric Functional Assessment

Standardized Patient Instructor

…into an OSCE

Karen E. Hall, M.D., Ph.D.Clinical Assistant Professor Division of Geriatric Medicine

University of Michigan, Ann Arbor VAMC


GFA SPI: CharacteristicsGFA SPI: Characteristics

• ““Essential Skill Set” for all physiciansEssential Skill Set” for all physicians

• Emphasis on function, not diseaseEmphasis on function, not disease

• Validated, rapid screening tools Validated, rapid screening tools

• FeedbackFeedback

• EducationEducation

• Multiple types and level of learnerMultiple types and level of learner


Steps to convert the SPI into an Steps to convert the SPI into an OSCEOSCE

1. Review scenario for reliability and validity1. Review scenario for reliability and validity (high stakes exam requires rigor)(high stakes exam requires rigor)

Is task is appropriate for level of learner and time allotted?Is task is appropriate for level of learner and time allotted? Err on side of “too little content” rather than “too much”Err on side of “too little content” rather than “too much”

Faculty review to ensure content is clinically relevant (validity)Faculty review to ensure content is clinically relevant (validity)

Review checklist items with SPs to ensure performance and Review checklist items with SPs to ensure performance and scoring is accurate. scoring is accurate.

Faculty should observe a percentage of OSCEs (at least 5-Faculty should observe a percentage of OSCEs (at least 5-10%) and score using same checklist as SPs (reliability)10%) and score using same checklist as SPs (reliability)


Case ScenarioCase Scenario

•85-year-old male or female85-year-old male or female

•Inpatient or Emergency RoomInpatient or Emergency Room

•““Ready for Discharge” BUT...Ready for Discharge” BUT...

•Functional impairmentFunctional impairment

•Social issue: caregiver stressSocial issue: caregiver stress

•Focus on functional assessment Focus on functional assessment

not diagnosisnot diagnosis


Administration of SPI in Learning Administration of SPI in Learning Resource Center (LRC)Resource Center (LRC)

Total time: 1 hour Minutes

Introduction 5SPI 20Learner Survey 5SPI feedback 10Faculty debrief 15

Learner evaluation of SPI 5


Administration of GFA OSCE in Administration of GFA OSCE in Learning Resource Center (LRC)Learning Resource Center (LRC)

Total time: Minutes

Review case on door 3SP OSCE 10

Learner self assessment 2

Entire CCA is 4 hours with 14 stations

Acute abdomen, Health beliefs, Chest pain, Chest pain note, Geriatric, Back pain, Pediatric, XRays/Imaging, EKG, Psychiatry, Evidence Based Medicine + 2 breaks


Type of Data available from OSCE

By Student:

• raw scores (not done=0,needs improvement=1, done=2) by item

• mean +/- SD of raw scores (“1.7 +/- 0.4”)

• percentage score (tasks; communication) – identifies pass/fail


Type of Data available from OSCE

Whole class:

• Histogram of scores by item (e.g. “done vs. “not done”) - Identifies deficiencies in class performance ? Curricular deficiency?

• Percentage for all stations “overall score” – distribution of the class: “high”; “average”; “low”


Educational GFA SPI:M1

0

25

50

75

100

Eatin

g

Transf

erin

g

Toiletin

g

Bathin

g

Dress

ing

Groom

ing

Per

cen

tag

e o

f M

1 st

ud

ents

Not Asked (1) Asked (2) Asked in Detail (3)

Evaluative GFA OSCE:M3

0

25

50

75

100

Per

cen

tag

e o

f M

3 S

tud

ents

Not Asked (0) Needs Improvement (1) Done (2)


OSCE: IADL Assessment

Students remembered to ask about meals and grocery shopping more frequently than medications. Students often neglected to ask in enough detail to score “done”.

0

25

50

75

100

Medications Prepare Meals Transportation Other

% o

f M

3 S

tud

ents

Not Asked (0) Needs Improvement (1) Done (2)


OSCE: Falls, Observe Gait, Timed Up and Go

Most students remember to ask about falls and observe gait, but few do objective test (TUG).

0

25

50

75

100

Asked about falls Observed Gait Timed Up and Go

% o

f M

3 S

tud

ents

Not Done (0) Needs Improvement (1) Done (2)


Special Communication

0

25

50

75

100

Facilitated Visual Faciliated Auditory

% o

f M

1 st

ud

ents

Poor (1) Adequate (2) Skillful (3)

0

25

50

75

100

Glasses Hearing

% o

f M

3 S

tud

ents

Not done (0) Needs Improvement (1) Done (2)

EducationalSPI

EvaluativeOSCE


Communication (9 items)

0

25

50

75

100

All Stations Geriatric

M1

Rat

ing

(1-

5 sc

ale)

Best performance (100%): “addressed by name”; “didn’t interrupt”Worst performance (60-62%): “asked open-ended questions”; ”summarized discussion”


GFA OSCE Data from PilotGFA OSCE Data from Pilot

1. Numerical percentage score on GFA

17 task items (max score =32)

Converted to percentage of 100%

Mean +/- SD for tasks 43 +/- 15%

Communication score 84 +/- 5 %


GFA OSCE Data from PilotGFA OSCE Data from Pilot

2. Learner self-evaluation score (% of 100) on GFA: “Estimate your average score for this station”

Mean +/- SD 74 +/- 12%

Clearly there is a discrepancy between the student’s assessment and their performance score of 43%!


Overall Performance Overall Performance (excluding GFA)(excluding GFA)

Overall score on CCA 76 +/- 6%

Passing average set at… 64%

Out of 162 students…..

62 students passed all stations

97 failed 1-3 stations – retake failed stations

3 students failed entire CCA

Significant correlation between failed 2 or more stations and very low GFA scores (less than 30) – overall lack of knowledge?


Hofstee Method to define Hofstee Method to define “pass/fail” for 2005 CCA“pass/fail” for 2005 CCA

“What is the lowest % students you would accept to fail station” - usually 0%

“What is the highest % students you would accept to fail station” - high for vital material, low for unimportant

“What is the lowest % cut-off you would accept to fail station” – low if hard material, high if easy

“What is the highest % cut-off you would accept to fail station” – high if vital


Hofstee Method to define cut Hofstee Method to define cut points for 2005 GFA OSCEpoints for 2005 GFA OSCE

As of March 2005 (AGS handouts due)….

Don’t know (still being analyzed – will know before May 2005) – Still don’t know!

Given poor performance and curriculum revision during 2004-2006: may need generous cutoff criteria

0%-40% failure rate, 45%-60% pass cut off

May be more like 10% failure, 50% cut off


Take home points for converting Take home points for converting an SPI to an OSCEan SPI to an OSCE

Keep it simple

Be prepared to justify your choice of material – to faculty and students (particularly those that fail!)

Make sure tasks are taught in curriculum!

Be prepared to review a lot of performances

Keep checking the reliability of the SPs

Fight off all attempts to change the OSCE checklist in mid-assessment (data will not be analyzable!)


Reference sitesReference sitesTalk will be on my website:

http://sitemaker.umich.edu/khallinfo

GFA Student manual and pocket card on POGOe (registration required):

http://www.pogoe.org/px/login.cfm

I can email other material if needed:

[email protected]

ags annual meeting may 2005 converting the geriatric functional assessment standardized patient...

Documents

ags annual meeting

level of learner slide

functional assessment

gfa spi osce

breaks slide

administration of spi

sps reliability slide

faculty review