alliance 2007 "best of conference" presentation and webinar: beyond theory: practical...
DESCRIPTION
My 2007 Alliance for CME conference presentation, later presented as an ACME webinar, that reviews strategies to measure continuing medical education outcomes. Was voted as a "Best of Conference" presentation, and included in the 2007 ACME “Best of Sessions Compilation CD.”TRANSCRIPT
Alliance for CME WebinarAlliance for CME WebinarTuesday, September 25, 2007Tuesday, September 25, 2007
Wendy Turell, DrPHSenior DirectorEducational Design and Accreditation ServicesBCME
Beyond Theory: Practical Beyond Theory: Practical Tools to Tackle Educational Tools to Tackle Educational
Outcomes EvaluationOutcomes Evaluation
22
Today’s AgendaToday’s Agenda
II Measurement and CMEMeasurement and CME
IIII Educational Outcomes Educational Outcomes Evaluation: Evaluation:
MethodologyMethodology
III III Sample Approaches to Study Sample Approaches to Study DesignDesign
IVIV Questions and AnswersQuestions and Answers
33
Rationale for MeasurementRationale for Measurement
Return on Education (ROE)Return on Education (ROE)– Educational dollars should be spent on Educational dollars should be spent on
effectiveeffective education education Guide for future programmingGuide for future programming
– What formats and strategies work best?What formats and strategies work best? Protection within CME guidelinesProtection within CME guidelines
– Proof of educational effectivenessProof of educational effectiveness
44
Evaluation and ACCME Evaluation and ACCME ComplianceCompliance
Criteria 11 Criteria 11
““The provider analyzes changes in learners The provider analyzes changes in learners (competence, performance, or patient (competence, performance, or patient
outcomes) achieved as a result of the overall outcomes) achieved as a result of the overall program’s activities/educational program’s activities/educational
interventions”interventions”
–Level 1: Provisional AccreditationLevel 1: Provisional Accreditation–Level 2: Full AccreditationLevel 2: Full Accreditation–Level 3: Accreditation with CommendationLevel 3: Accreditation with Commendation
An Outcomes-Based An Outcomes-Based Educational ModelEducational Model
66
Outcomes-Based Educational Outcomes-Based Educational ModelModel
Needs Assessment
Identify Optimal Outcomes (based on educational gap)
Outcomes Assessment & Program Evaluation
Outcomes-Based Learning Objectives
Design Program for Maximum Impact
Needs Assessment
77
Educational EvaluationEducational Evaluation
Tells us whether the planned goals match Tells us whether the planned goals match what learners take from the activitywhat learners take from the activity
Judges, describes, defines, values, shapes Judges, describes, defines, values, shapes opinion, directs attentionopinion, directs attention
Determines the value or quality of Determines the value or quality of educational activities in efforts to provide educational activities in efforts to provide feedback for improvement feedback for improvement – ““Value” can be in regards to: satisfaction, Value” can be in regards to: satisfaction,
achievement, improved performance, benefits achievement, improved performance, benefits to others, return on investment, etc.to others, return on investment, etc.
Bennett NL: The voices of evaluation. J Contin Educ Health Prof 1997;17:198-206.
ACME “Evaluating CME Using Outcomes” in Evaluating Educational Outcomes, http://www.acme-assn.org, accessed 1/2006.
Green, Joseph & Eckstein, Jane (2006) A Practical Guide to Integrating an Outcomes-Based Learning Model Into Your Planning Process, Alliance for CME Almanac, 28 (1), p 1-5.
88
Outcomes Evaluation Outcomes Evaluation
More Specific Type of Educational More Specific Type of Educational EvaluationEvaluation
Evaluation that strives to Evaluation that strives to demonstrate the relative demonstrate the relative effectiveness of various approaches effectiveness of various approaches to education on learning/behavior to education on learning/behavior change/patient health outcomes (a change/patient health outcomes (a causes b)causes b)
Bennett NL: The voices of evaluation. J Contin Educ Health Prof 1997;17:198-206.
ACME “Evaluating CME Using Outcomes” in Evaluating Educational Outcomes, http://www.acme-assn.org, accessed 1/2006.
Green, Joseph & Eckstein, Jane (2006) A Practical Guide to Integrating an Outcomes-Based Learning Model Into Your Planning Process, Alliance for CME Almanac, 28 (1), p 1-5.
99
Reach for Greater HeightsReach for Greater Heights in in
MeasurementMeasurement
LEVEL 5PATIENT HEALTH
LEVEL 4PERFORMANCE
LEVEL 3LEARNING
LEVEL 2SATISFACTION
LEVEL 1PARTICIPATION
LEVEL 6POPULATION HEALTH
Moore DE. A framework for outcomes evaluation. In: Davis D. Barnes BE, Fox R, eds. The Continuing Professional Development of Physicians: From Research to Practice. Chicago: AMA Press: 2003.
1010
Levels 1,2,3Levels 1,2,3
Levels 1: ParticipationLevels 1: Participation– Not a very valid assessment of Not a very valid assessment of educationaleducational outcome outcome– ObjectiveObjective
Level 2: SatisfactionLevel 2: Satisfaction– Does not document learning Does not document learning – SubjectiveSubjective
Level 3: LearningLevel 3: Learning– Assessment of educational outcomeAssessment of educational outcome– Can be measured with survey questionnairesCan be measured with survey questionnaires– May or may not lead to behavior changeMay or may not lead to behavior change
1111
Levels 4, 5 and 6Levels 4, 5 and 6
Level 4: PerformanceLevel 4: Performance– Various ways to measureVarious ways to measure– Can document impact on practice behavior using follow-up assessmentsCan document impact on practice behavior using follow-up assessments– May not capture all new behaviors May not capture all new behaviors – Typically self-reportedTypically self-reported
Level 5: Patient HealthLevel 5: Patient Health– Objective measure; desirable outcome variableObjective measure; desirable outcome variable– Difficult to determine if change is due to intervention; also clouded by Difficult to determine if change is due to intervention; also clouded by
patient co-morbiditiespatient co-morbidities– $$$$$, patient privacy laws (HIPPA) and Institutional Review Board Barriers$$$$$, patient privacy laws (HIPPA) and Institutional Review Board Barriers
Level 6: Population HealthLevel 6: Population Health– Most desirable outcome variable!Most desirable outcome variable!– Do most CME activities deliver such a reach??Do most CME activities deliver such a reach??– Rarely measured in CME due to financial and logistical barriersRarely measured in CME due to financial and logistical barriers
Outcomes Evaluation: Outcomes Evaluation: MethodologyMethodology
1313
Sample Study PlanSample Study Plan
Study Subjects Survey Administrati
on
Survey Design
Experimental
Group
Control Group
(optional)
Case-Based Questions
Knowledge- Based Questions Pre-Test
Staggered
Post-Test
Different options to assess learning and
behavior change
1414
Validity & Statistical Validity & Statistical SignificanceSignificance
ValidityValidity- The degree of confidence one can - The degree of confidence one can have in an observed result, such as an have in an observed result, such as an improvement in knowledgeimprovement in knowledge– The degree to which the observed result can be The degree to which the observed result can be
attributed to the studied cause (ie: effectiveness of attributed to the studied cause (ie: effectiveness of the CME course) and not random error in sampling the CME course) and not random error in sampling and measurementand measurement
Statistical SignificanceStatistical Significance – Quantifies the degree – Quantifies the degree of confidence you can have in a specific resultof confidence you can have in a specific result– Ex: Statistically significant at .05 alpha level = 95% Ex: Statistically significant at .05 alpha level = 95%
chance that the result is valid and not due to chance that the result is valid and not due to chancechance
1515
Some Ways to Increase ValiditySome Ways to Increase Validity
Choose participants fairly (everyone Choose participants fairly (everyone has an equal chance of completing has an equal chance of completing surveys)surveys)
Ensure robust sample sizeEnsure robust sample size Write a clear survey (participants Write a clear survey (participants
should understand questions as you should understand questions as you understand them)understand them)
Use a control group (placebo or Use a control group (placebo or comparison subjects)comparison subjects)
1616
Pre –Tests: Pre –Tests: Walking-in-the doorWalking-in-the door knowledgeknowledge
Pre-TestsPre-Tests- Establishes baseline of - Establishes baseline of subjects’ knowledge/behavior before subjects’ knowledge/behavior before they are exposed to our educational they are exposed to our educational intervention (EI)intervention (EI)– Ex: What do registrants for our activity Ex: What do registrants for our activity
know about the medical topic before know about the medical topic before they begin the course? How are they they begin the course? How are they treating patients with this condition?treating patients with this condition?
1717
Post –Tests: Post –Tests: Walking-out-the Walking-out-the doordoor knowledge knowledge
Post-TestsPost-Tests- Collect data on participant’s - Collect data on participant’s knowledge and clinical practice patterns knowledge and clinical practice patterns regarding the specific subject matter regarding the specific subject matter after after they are exposed to the EIthey are exposed to the EI– Ex: What do CME participants know about the Ex: What do CME participants know about the
medical topic after the course? How are they medical topic after the course? How are they treating patients with this condition now?treating patients with this condition now?
Staggered Post-Tests-Staggered Post-Tests- Administer the post- Administer the post-test several weeks or months later.test several weeks or months later.– Better capture of the true retention of knowledge Better capture of the true retention of knowledge
(or behavior change) by participants(or behavior change) by participants
1818
Experimental and Control Experimental and Control GroupsGroups
Experimental Group-Experimental Group- participants who are participants who are exposed to educational activitiesexposed to educational activities– We are interested in We are interested in theirtheir learning and behavior learning and behavior
change outcomeschange outcomes
Control Group-Control Group- A “comparison” group of A “comparison” group of similar individuals who are not exposed to similar individuals who are not exposed to learning activitieslearning activities– We are interested in how the experimental We are interested in how the experimental
subjects’ outcomes differ from the control subjects’ outcomes differ from the control subjects’ outcomessubjects’ outcomes
– ““Match” on demographic variablesMatch” on demographic variables
1919
Survey Question TypesSurvey Question Types
Knowledge-BasedKnowledge-Based
Practice BasedPractice Based– Case StudiesCase Studies
2020
Sample “Practice-Based” Case Sample “Practice-Based” Case Study QuestionStudy Question
GT is a 73 year-old, non-smoking woman GT is a 73 year-old, non-smoking woman with no history of urinary complaints, with no history of urinary complaints, pelvic organ prolapse or comorbid pelvic organ prolapse or comorbid conditions. GT presents with primary conditions. GT presents with primary complaints of urgency and urinary leakage. complaints of urgency and urinary leakage. She states that she cannot travel beyond She states that she cannot travel beyond her immediate neighborhood, or attend her immediate neighborhood, or attend social events, for fear of experiencing social events, for fear of experiencing episodes of urinary incontinence. Her episodes of urinary incontinence. Her primary goal is to control her symptoms primary goal is to control her symptoms and regain a normal lifestyle.and regain a normal lifestyle.
2121
Sample Case Study Sample Case Study QuestionQuestion
What steps would you initiate in the evaluation of GT?What steps would you initiate in the evaluation of GT?a.a. Recommend initiation of a bladder diaryRecommend initiation of a bladder diaryb.b. Utilize a questionnaire with GT to help distinguish between urge and/or Utilize a questionnaire with GT to help distinguish between urge and/or
stress incontinencestress incontinencec.c. Perform a urinalysis Perform a urinalysis d.d. All of the aboveAll of the above
Which of the following steps would you take in the management of GT’s Which of the following steps would you take in the management of GT’s condition?condition?
a.a. Teach the patient to initiate pelvic floor muscle exercisesTeach the patient to initiate pelvic floor muscle exercisesb.b. Educate the patient to avoid bladder irritantsEducate the patient to avoid bladder irritantsc.c. Both A & BBoth A & Bd.d. Sacral nerve modulationSacral nerve modulation
Which of the following is LEAST likely to be a diagnosis for GT?Which of the following is LEAST likely to be a diagnosis for GT?a.a. Overactive bladderOveractive bladderb.b. Bladder outlet obstruction Bladder outlet obstruction c.c. Urinary tract infectionUrinary tract infectiond.d. Stress urinary incontinenceStress urinary incontinence
2222
Sample SizeSample Size
You don’t have to poll everyone!You don’t have to poll everyone! Effect SizeEffect Size Power CalculationPower Calculation How to Estimate without having a How to Estimate without having a
statistics degreestatistics degree Rules of Thumb: If you have <30 Rules of Thumb: If you have <30
participants, try to sample all of themparticipants, try to sample all of them
2323
IncentivesIncentives– Encourage participation Encourage participation
We want to reach our target “n”We want to reach our target “n”
– Should be “medically relevant” Should be “medically relevant”
– They They areare permitted in CME permitted in CME AMA guidelines- AMA guidelines- << $100.00 $100.00 To avoid the perception of coercion, may wish To avoid the perception of coercion, may wish
to stay lower to stay lower
– Resources:Resources: www.medicalbooks.comwww.medicalbooks.com www.medbookstore.comwww.medbookstore.com www.allheart.comwww.allheart.com http://solutions.medsite.com/medsite_rewards.asphttp://solutions.medsite.com/medsite_rewards.asp
2424
Challenges to MethodologyChallenges to Methodology
Obtaining contact information for pretest Obtaining contact information for pretest contactcontact– LiveLive– Enduring MaterialsEnduring Materials
Funding LimitationsFunding Limitations
Grantor concernsGrantor concerns– Pre-testsPre-tests– IncentivesIncentives– ObjectivityObjectivity
2525
Challenges to MethodologyChallenges to Methodology
??
2626
Sample ApproachesSample Approaches
Live Meeting (i.e.: Symposium)Live Meeting (i.e.: Symposium)– 250 attendees anticipated250 attendees anticipated– Obtained e-mails via pre-registration Obtained e-mails via pre-registration
websitewebsite Linked pre-test to this siteLinked pre-test to this site
– E-mailed post-test to pre-registrants 4 E-mailed post-test to pre-registrants 4 weeks after the live meetingweeks after the live meeting Screener question to assure they attended Screener question to assure they attended
activityactivity Match pre-test and post-test resultsMatch pre-test and post-test results
2727
Sample ApproachesSample Approaches
Journal SupplementJournal Supplement– Distribution to 7,000 physiciansDistribution to 7,000 physicians
Publisher could not provide e-mail addresses of Publisher could not provide e-mail addresses of recipientsrecipients
Could not anticipate “readers” from population of Could not anticipate “readers” from population of 7,0007,000
– Administered pre-test to control group of Administered pre-test to control group of similarly specialized physicianssimilarly specialized physicians
– Administered post-test to readers and control Administered post-test to readers and control goupgoup Recruited via mention of post-test and incentive in Recruited via mention of post-test and incentive in
the rear of printed supplementthe rear of printed supplement
2828
Sample ApproachesSample Approaches
Online CME ActivityOnline CME Activity– Challenge = no funding for incentivesChallenge = no funding for incentives– 1,000 anticipated participants1,000 anticipated participants– Present with optional (opt-out) pre-test Present with optional (opt-out) pre-test
during log-in/registrationduring log-in/registration– Present with opt-out post-test at close of Present with opt-out post-test at close of
activityactivity
2929
Sample ApproachesSample Approaches
PodcastPodcast– 500 downloads anticipated500 downloads anticipated– Obtained e-mails at point of Obtained e-mails at point of
download/registrationdownload/registration– Opt-out pre-test offered at point of Opt-out pre-test offered at point of
downloaddownload– Mention URL (easy to recall address) at Mention URL (easy to recall address) at
close of podcastclose of podcast mention incentivemention incentive
3030
Example 1Example 1 You wish to evaluate the educational You wish to evaluate the educational
outcomes of a live meeting regarding Diabetes outcomes of a live meeting regarding Diabetes that is to be held next month in your hospital. that is to be held next month in your hospital. The meeting is targeting physicians and The meeting is targeting physicians and nurses, and you anticipate a turnout of 100 nurses, and you anticipate a turnout of 100 participants. Although your funding is limited, participants. Although your funding is limited, you do have unlimited access to the hospital you do have unlimited access to the hospital information technology department should information technology department should you need computer programming assistance.you need computer programming assistance.
– How would you approach assessing outcomes How would you approach assessing outcomes for this CME Activity?for this CME Activity?
3131
Example 2Example 2 Your organization has received funding for a Your organization has received funding for a
series of 6 podcasts on the topic of Heart series of 6 podcasts on the topic of Heart Failure. Each of the 6 podcasts will feature Failure. Each of the 6 podcasts will feature different thought leader interviews regarding different thought leader interviews regarding hot topics in the therapeutic area. You are hot topics in the therapeutic area. You are not sure how many participants will listed to not sure how many participants will listed to your podcasts, since this is the first time your podcasts, since this is the first time your company has administered this type of your company has administered this type of activity. You have a budget of $10,000 for activity. You have a budget of $10,000 for “outcomes” for the entire series. “outcomes” for the entire series.
– How would you approach assessing How would you approach assessing outcomes for this CME Activity?outcomes for this CME Activity?
3232
Example 3Example 3 Your company has received funding for a Your company has received funding for a
monograph publication that is based on the monograph publication that is based on the proceedings of a live event on the topic of proceedings of a live event on the topic of Alzheimer’s Disease. The publishing Alzheimer’s Disease. The publishing company informs you that the monograph company informs you that the monograph will be distributed to 9,000 physicians and will be distributed to 9,000 physicians and allied health care professionals. You have allied health care professionals. You have $15,000 to perform an outcomes assessment $15,000 to perform an outcomes assessment of the enduring material.of the enduring material.
– How would you approach assessing How would you approach assessing outcomes for this CME Activity?outcomes for this CME Activity?
3333
Questions and AnswersQuestions and Answers
???
3434
Contact InformationContact Information
Wendy Turell, DrPHWendy Turell, DrPH
Associate Vice President, Educational Associate Vice President, Educational Design and Accreditation ServicesDesign and Accreditation Services
BCMEBCME
[email protected]@bcmeonline.com
www.bcmeonline.comwww.bcmeonline.com