development and implementation of a log-based assessment tool for clinical clerkship

Development and Development and Implementation of a Log-Implementation of a Log-based Assessment Tool based Assessment Tool for Clinical Clerkshipfor Clinical Clerkship

Susan Chamberlain, MD, FRCSCSusan Chamberlain, MD, FRCSC

Phil Hahn, MScPhil Hahn, MSc

Dept of Obstetrics & GynaecologyDept of Obstetrics & Gynaecology

Health Sciences Education RoundsHealth Sciences Education Rounds

May 21, 2009May 21, 2009

ObjectivesObjectives

Describe log system linked to Describe log system linked to assessment and its rationaleassessment and its rationale

Present results of survey of Present results of survey of Meds2009 regarding their Meds2009 regarding their attitudes towards their attitudes towards their clinical logs (quantitative and clinical logs (quantitative and qualitative data)qualitative data)

BackgroundBackground

Problem # 1Problem # 1: :

ED-2 requires students in clerkship to ED-2 requires students in clerkship to log clinical experiences but student log clinical experiences but student participation is lowparticipation is low Students are not altruisticStudents are not altruistic Logs are incomplete and inaccurateLogs are incomplete and inaccurate

If results will be used to direct If results will be used to direct courses, they must be accuratecourses, they must be accurate

Students will only provide accurate Students will only provide accurate information, if there is clear information, if there is clear individual benefitindividual benefit


Problem # 2Problem # 2: :

Ob/Gyn end of rotation SAQ examOb/Gyn end of rotation SAQ exam Out of dateOut of date Time consuming to mark and Time consuming to mark and maintainmaintain

Inconsistency in markingInconsistency in marking


January 2006 (Meds 2007) – new January 2006 (Meds 2007) – new system of evaluation linked to system of evaluation linked to clinical log startedclinical log started Each logged encounter corresponded to Each logged encounter corresponded to LMCC clinical presentationLMCC clinical presentation

Each covered by clerkship seminarEach covered by clerkship seminar Students logged patient encounters, Students logged patient encounters, seminars attendedseminars attended

Exam question chosen based on clinical Exam question chosen based on clinical log (i.e. students only examined in SOE log (i.e. students only examined in SOE on what they had seen)on what they had seen)


MCQ lengthenedMCQ lengthened SAQ replaced by Structured Oral SAQ replaced by Structured Oral examination (SOE)examination (SOE) 50 mark questions focusing on data gathering 50 mark questions focusing on data gathering (history, physical and choice of (history, physical and choice of investigation, plus basic management choices)investigation, plus basic management choices)

Each exam question written by attending staff Each exam question written by attending staff who gives the seminar (all reviewed/edited by who gives the seminar (all reviewed/edited by course chair for consistency)course chair for consistency)

Student answers 2 questions in 30 minute examStudent answers 2 questions in 30 minute exam

Survey on Student Survey on Student AttitudesAttitudesMarch 2008March 2008 Survey of class of meds 2009, blocks Survey of class of meds 2009, blocks 2-8 on use of logs, attitudes2-8 on use of logs, attitudes

Student asked to complete online Student asked to complete online questionnaire after 2 clerkship questionnaire after 2 clerkship rotations: ob/gyn and surgeryrotations: ob/gyn and surgery

Confidentiality of responses assured Confidentiality of responses assured by code to identify responsesby code to identify responses

74 student surveyed74 student surveyed

Survey ResultsSurvey Results 34 Clerks filled out questionnaire 34 Clerks filled out questionnaire after one of the rotations (OB or after one of the rotations (OB or Surgery)Surgery)

19 Clerks filled out questionnaire 19 Clerks filled out questionnaire after both rotations (paired data)after both rotations (paired data) Responses after first rotation (either Responses after first rotation (either OB or Surgery) were included in the OB or Surgery) were included in the analysisanalysis

Survey ResultsSurvey Results

CodesCodes One duplicate One duplicate (0023f91973) (0023f91973) (Block 2 OB and Block 3 Surgery)(Block 2 OB and Block 3 Surgery)

One with no match One with no match (a0f3d5c663)(a0f3d5c663)

These two responses were not These two responses were not included in analysisincluded in analysis

Survey ResultsSurvey Results 53 Clerks53 Clerks Two independent groupsTwo independent groups

29 OB rotation29 OB rotation 24 Surgery rotation24 Surgery rotation

Q1. I completed my log:Q1. I completed my log:

After each patientAfter each patient DailyDaily 1-2 times/week1-2 times/week Less than once/weekLess than once/week


ResponseResponseRotationRotation

OB * OB * (n=28)(n=28)

SurgerySurgery(n=24)(n=24)

After each patient After each patient encounterencounter

0 (0)0 (0) 0 (0)0 (0)

DailyDaily 2 (7.1)2 (7.1) 0 (0)0 (0)

1-2 times per week1-2 times per week 14 (50.0)14 (50.0) 5 (20.8)5 (20.8)

Less than once per Less than once per weekweek

12 (42.9)12 (42.9) 19 (79.2)19 (79.2)* One response (both 1-2 times per week and less than once per week) not included

Values = n (%)



OB OB (n=28)(n=28)


After each patient After each patient encounterencounter

0 (0)0 (0) 0 (0)0 (0)

DailyDaily 2 (7.1)2 (7.1) 0 (0)0 (0)

1-2 times per week1-2 times per week 14 (50.0)14 (50.0) 5 (20.8)5 (20.8)


12 (42.9)12 (42.9) 19 (79.2)19 (79.2)For analysis, “daily” and “1-2 times per week” categories were pooled.

Values = n (%)



OB OB (n=28)(n=28)


1-2 times per week / 1-2 times per week / dailydaily

16 (57.1)16 (57.1) 5 (20.8)5 (20.8)


12 (42.9)12 (42.9) 19 (79.2)19 (79.2)Values = n (%)

P = 0.011, Fisher’s exact test (statistically significant)

Clerks logged more frequently during the OB rotation.

Q2. I found logging useful to Q2. I found logging useful to my learning.my learning.

YesYes NoNo

Q2. I found logging useful to Q2. I found logging useful to my learning.my learning.

OB: 20.7% (6/29)OB: 20.7% (6/29) Surgery: 12.5% (3/24)Surgery: 12.5% (3/24)

Difference = 8.2%Difference = 8.2% P = 0.487, Fisher’s exact test P = 0.487, Fisher’s exact test

Q3. I logged some clinical Q3. I logged some clinical encounters that I encounters that I did notdid not see.see.

YesYes NoNo

Q3. I logged some clinical Q3. I logged some clinical encounters that I encounters that I did notdid not see.see.



Q4. I chose Q4. I chose not to log not to log some some clinical problems that I clinical problems that I did see.did see.

YesYes NoNo

Q4. I chose Q4. I chose not to log not to log some some clinical problems that I clinical problems that I did see.did see.



Q5. After logging clinical Q5. After logging clinical encounter, I did encounter, I did further reading on the topic further reading on the topic ___% of the time___% of the time

Q5: After logging clinical Q5: After logging clinical encounter, I did further encounter, I did further

reading on the topic:reading on the topic:% ofClerks

Response: % of the time.

OB Rotation (n=28), 1 missing

Median (Middle) Response = 15% of the time

Q5: After logging clinical Q5: After logging clinical encounter, I did further encounter, I did further

reading on the topic:reading on the topic:

% ofClerks

Response: % of the time.

Surgery Rotation (n=23), 1 missing

Median Response = 10% of the time

Q5. After logging clinical Q5. After logging clinical encounter, I did encounter, I did

further reading on the further reading on the topic:topic:

Median (% of the time)Median (% of the time) OB: 15% versus Surgery: 10% OB: 15% versus Surgery: 10%

P = 0.453, Mann-Whitney U testP = 0.453, Mann-Whitney U test(nonparametric version of the t test) (nonparametric version of the t test)

Q6: Average time (minutes) Q6: Average time (minutes) spent reading on an clinical spent reading on an clinical encounter encounter

Q6: Average time (minutes) Q6: Average time (minutes) spent reading on an clinical spent reading on an clinical

encounter encounter

% ofClerks

Time (minutes)

OB Rotation (n=28), 1 missing

Median = 30 minutes

Q6: Average time (minutes) Q6: Average time (minutes) spent reading on an clinical spent reading on an clinical

encounter encounter

% ofClerks

Time (minutes)

Surgery Rotation (n=22), 2 missing

Median = 30 minutes

Q6. Average time (minutes) Q6. Average time (minutes) spent reading on an spent reading on an clinical encounter clinical encounter

Median (minutes)Median (minutes) OB: 30 versus Surgery: 30OB: 30 versus Surgery: 30

P = 0.935, Mann-Whitney U testP = 0.935, Mann-Whitney U test(nonparametric version of the t test) (nonparametric version of the t test)

Q7. Logging prompted me to Q7. Logging prompted me to read more about patients read more about patients that I had seen.that I had seen.

YesYes NoNo

Q7. Logging prompted me to Q7. Logging prompted me to read more about patients read more about patients

that I had seen.that I had seen.



Q8. Logging clinical Q8. Logging clinical encounters helped focus my encounters helped focus my studying.studying.

YesYes NoNo

Q8. Logging clinical Q8. Logging clinical encounters helped focus my encounters helped focus my

studying.studying.



SummarySummary

QuestionQuestionObstetricObstetric

ss(n=29)(n=29)


DifferenceDifference(OB - (OB -

Surgery)Surgery)P valueP value

Q2.Completed log Q2.Completed log 1-2 1-2 times per wk or times per wk or daily daily

16/28 16/28 57.1%57.1%

5 5 20.8%20.8% 36.3%36.3% 0.0110.011

Q3.Logging usefulQ3.Logging useful

6 6 20.7%20.7%

3 3 12.5%12.5% 8.2%8.2% 0.4870.487

Q4.Logged Q4.Logged encounters not encounters not seenseen

3 3 10.3%10.3%

1 1 4.2%4.2% 6.1%6.1% 0.6170.617

Q5.Didn’t log Q5.Didn’t log encounters seenencounters seen

18 18 62.1%62.1%

15 15 62.5%62.5%

(minus)(minus)- 0.4%- 0.4% 1.0001.000

Q8.Read moreQ8.Read more 226.9%6.9%

3312.5%12.5%

(minus)(minus)-5.6%-5.6% 0.6490.649

Q9.Helped focus Q9.Helped focus studiesstudies

6620.7%20.7%

114.2%4.2% 16.5%16.5% 0.1120.112

Fisher’s exact test

SummarySummary

QuestionQuestionObstetricsObstetrics(N=29)(N=29)

SurgerySurgery(N=24)(N=24) P valueP value

Q6.Futher reading Q6.Futher reading (% of the time)(% of the time)

15%15%(0%, 91%)(0%, 91%)

10%10%(0%, 74%)(0%, 74%)

0.4530.453

Q7.Time spent reading Q7.Time spent reading (minutes) (minutes)

3030(0, 120)(0, 120)

3030(0, 111)(0, 111)

0.6490.649

Mann-Whitney U testnonparametric version of the t testValues are medians (5th, 95th percentiles)

Q6. One missing value in each groupQ7. One missing value in OB group and two missing values in surgery group.

Conclusions Conclusions (Quantitative data)(Quantitative data) There is no difference with regards to There is no difference with regards to study use of the log or students’ study use of the log or students’ perception of usefulness of logs as an perception of usefulness of logs as an educational tool between these 2 educational tool between these 2 rotationsrotations

Students on the ob/gyn rotation logged Students on the ob/gyn rotation logged more frequentlymore frequently

A minority of students find that A minority of students find that logging is useful to their learninglogging is useful to their learning

Conclusions – accuracy Conclusions – accuracy of logsof logs Up to 10% of students logged Up to 10% of students logged clinical problems that they did not clinical problems that they did not seesee

62% of students are not logging the 62% of students are not logging the clinical problems that they are clinical problems that they are seeingseeing

Conclusions – log as Conclusions – log as study toolstudy tool Few students use the log as an Few students use the log as an impetus to read around patient impetus to read around patient encountersencounters

Few students find the log helpful as Few students find the log helpful as a focus for their studya focus for their study

Survey Results – Survey Results – qualitative dataqualitative data

Q9: Please comment on how Q9: Please comment on how clinical encounter logs may clinical encounter logs may have contributed to your have contributed to your education in either a positive education in either a positive or negative way.or negative way.

Results – Results – technical/communication technical/communication issuesissues Students find the interface difficult Students find the interface difficult and are frustrated by the limited and are frustrated by the limited choices, what ‘counts’ as a clinical choices, what ‘counts’ as a clinical encounterencounter ““too many drop-down menus”too many drop-down menus” ““topics were not arranged in a useful way”topics were not arranged in a useful way” ““cumbersome”cumbersome” ““... told by staff that you don’t have to ... told by staff that you don’t have to log everything, then get in trouble when log everything, then get in trouble when not everything is logged.”not everything is logged.”

Results – logging is a Results – logging is a ‘make-work project’‘make-work project’ Students consider the log ‘an Students consider the log ‘an administrative task which has administrative task which has no bearing’ on learningno bearing’ on learning

““felt like a tedious waste of time”felt like a tedious waste of time” ““..waste of time that could have been ..waste of time that could have been used studying..”used studying..”

……

Results – Results – Honesty/accuracy of the Honesty/accuracy of the logslogs Unclear thresholds for logging Unclear thresholds for logging of clinical activityof clinical activity

““sometimes difficult to know what counts sometimes difficult to know what counts as having ‘seen’ a clinical problem”as having ‘seen’ a clinical problem”

“…“…I logged...only those patients that I I logged...only those patients that I actually assessed…provides the greatest actually assessed…provides the greatest learning”learning”

Results – Results – honesty/accuracyhonesty/accuracy

Inaccurate recollection Inaccurate recollection

““I was inconsistent in filling logs I was inconsistent in filling logs out, sometimes forgetting for up to a out, sometimes forgetting for up to a week”week”

““I was fairly reliable about logging I was fairly reliable about logging for the first 5 blocks, then completely for the first 5 blocks, then completely forgot about it for the last 3”forgot about it for the last 3”

Results – Results – honesty/accuracy honesty/accuracy Students logged encounters they did Students logged encounters they did not seenot see ““..often completing logs at the conclusion of ..often completing logs at the conclusion of a rotation by simply making up cases that a rotation by simply making up cases that would allow me to cover rotation objectives”would allow me to cover rotation objectives”

“ “ does not ensure anything as it can easily does not ensure anything as it can easily be faked”be faked”

““..students may be tempted to log encounters ..students may be tempted to log encounters they have not experienced to avoid such they have not experienced to avoid such emails”emails”

Results – Results – honesty/accuracyhonesty/accuracy Students did not log encounters Students did not log encounters they did in fact seethey did in fact see

““I did choose to not log a specific I did choose to not log a specific encounter because I did not want to be encounter because I did not want to be asked the question on my oral exam as I asked the question on my oral exam as I did not have the chance to fully read did not have the chance to fully read around the topic”around the topic”

Results – study habitsResults – study habits

Some students found the log useful Some students found the log useful for studying:for studying:

““It does cause me to reflect on particular It does cause me to reflect on particular cases but fitting this extra task in my cases but fitting this extra task in my daily schedule feels like adding one more daily schedule feels like adding one more juggling ball to a performance that exists juggling ball to a performance that exists on the edge of completely falling apart”on the edge of completely falling apart”

““It probably did not increase my reading of It probably did not increase my reading of topics I did encounter, but would likely topics I did encounter, but would likely make me focus on those topics I hadn’t make me focus on those topics I hadn’t encountered”encountered”

Results – study habitsResults – study habits

Most students did not find the Most students did not find the log useful for studying:log useful for studying:

““I read about things that I did not I read about things that I did not understand as they came up, rather than understand as they came up, rather than after I had logged them.”after I had logged them.”

““Logging was not a motivation for me to Logging was not a motivation for me to read more, seeing my patients was.”read more, seeing my patients was.”

Results – Q9Results – Q9

““.. I haven’t found it helpful for identifying .. I haven’t found it helpful for identifying areas of deficiencies. For example I went through areas of deficiencies. For example I went through 6 weeks of surgery (incl urol) without having a 6 weeks of surgery (incl urol) without having a single opportunity to put in a foley catheter”single opportunity to put in a foley catheter”

“… “… after the mid-way meeting and outside of after the mid-way meeting and outside of ob/gyn, no one else seemed to give a toss about ob/gyn, no one else seemed to give a toss about the log, so I didn’t either”the log, so I didn’t either”

““these .. would be more beneficial if they were these .. would be more beneficial if they were emphasized by the surg/obsgyn teams during clerk emphasized by the surg/obsgyn teams during clerk rotations, with some minor teaching involved…rotations, with some minor teaching involved…

Results – positive Results – positive commentscomments Ensured rotation objectives met:Ensured rotation objectives met:

““they direct you to see out certain they direct you to see out certain clinical opportunities that you haven’t clinical opportunities that you haven’t had yet”had yet”

““Did not directly affect my learning for Did not directly affect my learning for the block, but did ensure that I had the block, but did ensure that I had seen all the objectives laid out for the seen all the objectives laid out for the block”block”


Q10: With respect to the OB/GYN Q10: With respect to the OB/GYN rotation, how did knowing that the rotation, how did knowing that the topics for your final oral exam topics for your final oral exam were selected after reviewing your were selected after reviewing your clinical encounter log:clinical encounter log:

A: Influence your logging habits?A: Influence your logging habits? B: Contribute to your overall B: Contribute to your overall learning experience?learning experience?


Q10: With respect to the OB/GYN Q10: With respect to the OB/GYN rotation, how did knowing that rotation, how did knowing that the topics for your final oral the topics for your final oral exam were selected after exam were selected after reviewing your clinical reviewing your clinical encounter log:encounter log:

A: Influence your logging A: Influence your logging habits?habits?

Survey Results – Q10ASurvey Results – Q10A

Some examiners did not comply Some examiners did not comply with guidelines and asked about with guidelines and asked about topics not loggedtopics not logged

Survey results – Q10ASurvey results – Q10A

Obs/gyn students had a higher Obs/gyn students had a higher threshold for loggingthreshold for logging

““It made me be really sure I adequately It made me be really sure I adequately ‘saw’ something before I logged it”‘saw’ something before I logged it”

““.. Didn’t log things only seen briefly .. Didn’t log things only seen briefly that I didn’t want to be tested on”that I didn’t want to be tested on”

Survey results – Q10ASurvey results – Q10A

Some students more Some students more conscientious on obs/gyn conscientious on obs/gyn rotationrotation

““I was more complete in my logs than on I was more complete in my logs than on previous rotations”previous rotations”

“…“…More of an impetus to log the things More of an impetus to log the things I had seen”I had seen”


Q10: With respect to the OB/GYN Q10: With respect to the OB/GYN rotation, how did knowing that rotation, how did knowing that the topics for your final oral the topics for your final oral exam were selected after exam were selected after reviewing your clinical reviewing your clinical encounter log:encounter log:

B: Contribute to your overall B: Contribute to your overall learning experience?learning experience?

Survey results – Q10BSurvey results – Q10B

Focusing/narrowing exam studyFocusing/narrowing exam study Yes –Yes –

helped focus my studying for the oral examhelped focus my studying for the oral exam It was nice to see that the logs were actually being It was nice to see that the logs were actually being usedused

If anything the log-exam relationship gave the student If anything the log-exam relationship gave the student an excuse to not study certain areasan excuse to not study certain areas

No –No – the objectives on the log are too vaguethe objectives on the log are too vague Was surprised at how intense and wide spread the oral Was surprised at how intense and wide spread the oral exam questions are. I think it’s just safest to study exam questions are. I think it’s just safest to study everything:)everything:)

I had to know it all for the MCQ anywayI had to know it all for the MCQ anyway

Lessons learnedLessons learned

Although the students logged more Although the students logged more regularly on ob/gyn they do not regularly on ob/gyn they do not appear to log with more accuracyappear to log with more accuracy

Although students appreciate that Although students appreciate that the log is demonstrably used in some the log is demonstrably used in some way, the link between log and way, the link between log and evaluation does not add any apparent evaluation does not add any apparent educational benefiteducational benefit

Lessons learnedLessons learned

The current system and interface The current system and interface is still problematic and may be is still problematic and may be preventing the log from attaining preventing the log from attaining its potential as a recording toolits potential as a recording tool

Faculty and residents need to be Faculty and residents need to be more aware of the students’ more aware of the students’ objectivesobjectives

Future research ?Future research ?

Future research ?Future research ?

““I found that focusing on I found that focusing on studying in a PROBLEM-BASED studying in a PROBLEM-BASED manner (i.e. to be able to manner (i.e. to be able to discuss my clinical approach in discuss my clinical approach in the oral exam) was VERY the oral exam) was VERY EFFECTIVE.”EFFECTIVE.”

QUESTIONS? COMMENTS?QUESTIONS? COMMENTS?

development and implementation of a log-based assessment tool for clinical clerkship

Documents

clinical log startedeach

responserotationob n

rotations ob

objectivesdescribe log

clinical experiences

logbased assessment

clinical logs quantitative

clerkship rotations