100518 presentation final - webinars, webcasts, lms...
TRANSCRIPT
Polling Question
How many people are participating in this webinar at your location today?
Just me! 72 83 94 105 More than 106
Polling Question
What member section do you belong to?
Health Care Education AssociationFederal/Government Health Care Educator
Hospital/Health SystemMedical Education and Communication Company
Medical SchoolMedical Specialty Society
State Medical SocietyPharmaceutical
Other
Education Outcomes Assessment for
Regularly Scheduled Series (RSS)
Tuesday, May 18, 20102:00 – 3:00 PM ET
Presenter
Jason Olivieri, MPHMedical Education Outcome Analyst
Nemours Foundation
I have no commercial relationships to disclose
CME OutcomesExpanded CME Framework Description Source of Data
Satisfaction (Level 2) The degree to which the expectations of the participants about the setting and delivery of the CME activity were met
Questionnaires completed by attendees after a CME activity
Declarative knowledge (Level 3A) Degree to which participants state what the CME activity intended them to know
Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain
Procedural knowledge (Level 3B) Degree to which participants state how to do what the CME activity intended them to know how to do
Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain
Competence (Level 4) Degree to which participants showin an educational setting how to do what the CME activity intended them to be able to do
Objective : Observation in educational settingSubjective : Self-report of competence; intention to change
Performance (Level 5) Degree to which participants do what the CME activity intended them to be able to do in their practices
Objective : Observation of performance in patient care setting; patient charts; administrative databasesSubjective : Self-report of performance
Moore DE, et al. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof 2009;29:1-15.
Interpreting a level 4+ CME outcome
• Physician participants in this RSS are
200% more likely to treat their diabetic patients in accordance with the latest
clinical guidelines
• Why? What was it about the activity that led to its success? How did the participants perceive it?
RSS
• Occur as frequently as once-a-week
• Types:
– Single discipline (e.g., Anesthesia Case
Conference)
– Multidisciplinary (e.g., Grand Rounds)
• You don’t have to evaluate every session
Webinar pre survey: How many RSS do you oversee? (n = 52)
0
20
40
60
80
100
120
140
160
180
# of RSS
Median = 20, range = 0 – 167!
RSS evaluation goals
• Assess the activity for satisfying criteria associated with “good” CME
• Assess participants for changes in competency, performance or patient health (ACCME, Criteria 11)
• Identify practice gaps (ACCME, Criteria 2)
• Increase response rate
Webinar pre survey: How do you
currently evaluate your RSS? (n = 52)
• ARS• Chart review
• Commitment to change evaluation• Designated evaluators• Focus groups
• Paper evaluations (post, quarter, annual)• Pre-post surveys• QI data
• Web-based survey
Where to start?
• 0% of 10 CME evaluations between 2000-2006 using self-developed instruments provided reliability or validity information
• J Contin Educ Health Prof 2007;27:16-27
Established CME assessment tools
1. Rothman AI, Sibbald G. Evaluating Medical Grand Rounds. J Contin Educ Health Prof 2002;22:77-83.
2. Wood TJ, et al. The development of a participant questionnaire to assess continuing medical education presentations. Med Educ2005;39:568-72.
3. Shewchuk RM, et al. A Standardized Approach to Assessing Physician Expectations and Perceptions of Continuing Medical Education. J Contin Educ Health Prof 2007;27:173-82.
CME assessment tool #1: Rothman & Sibbald*
• Rate presenters on ten items reflecting “best educational practice”
• 5-point scale
*J Contin Educ Health Prof 2002;22:77-83.
1. Provided objectives
2. Demonstrated thorough knowledge
3. Clear and organized
4. Stimulated enthusiasm
5. Appropriate level and detail
6. Effective visuals
7. Effective presentation style
8. 25% session interactive
9. Good rapport with audience
10. Invited questions and participation
CME assessment tool #2: Wood et al*
• Rate quality of CME presentation in three domains (nine items total)
• 7-point scale
*Med Educ 2005;39:568-72.
Domain 1: Presenter
1. Enthusiasm
2. Interaction with audience
3. Apparent knowledge of topic
Domain 2: Presentation
4. Information was presented in an organized manner
5. Used case-based methods; related information presented to practical problems
6. Quality of audiovisual
Domain 3: Content
7. Volume and complexity of the information was appropriate
8. Related content to current evidence in literature
9. Content was relevant to practice
CME assessment tool #3: Shewchuk et al*
• Assess participant perceptions of CME quality on ten items
• 7-point scale
*J Contin Educ Health Prof 2007;27:173-82.
1. Addressed my most pressing concerns
2. Addressed competencies identified by my specialty
3. Provided fair and balanced content
4. Provided clear evidence to support content
5. Included opportunities to learn interactively
6. Provided me with supporting materials or tools
7. Included opportunities to solve cases relevant to my work
8. Translated evidence to practice
9. Addressed barriers to optimal performance
10. Allowed me to assess what I have learned
CME OutcomesExpanded CME Framework Description Source of Data
Satisfaction (Level 2) The degree to which the expectations of the participants about the setting and delivery of the CME activity were met
Questionnaires completed by attendees after a CME activity
Declarative knowledge (Level 3A) Degree to which participants state what the CME activity intended them to know
Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain
Procedural knowledge (Level 3B) Degree to which participants state how to do what the CME activity intended them to know how to do
Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain
Competence (Level 4) Degree to which participants showin an educational setting how to do what the CME activity intended them to be able to do
Objective : Observation in educational settingSubjective : Self-report of competence; intention to change
Performance (Level 5) Degree to which participants do what the CME activity intended them to be able to do in their practices
Objective : Observation of performance in patient care setting; patient charts; administrative databasesSubjective : Self-report of performance
Moore DE, et al. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof 2009;29:1-15.
Competency outcome (criteria 11)
Do you intend to make any changes in your practice Do you intend to make any changes in your practice Do you intend to make any changes in your practice Do you intend to make any changes in your practice as a result of participating in this CME activity?as a result of participating in this CME activity?as a result of participating in this CME activity?as a result of participating in this CME activity?
□ If yes, please provide at least one example _____________
□ If no, please specify why not ________________________
Practice gaps (criteria 2)
1.1.1.1. What questions in practice are you having that you are not What questions in practice are you having that you are not What questions in practice are you having that you are not What questions in practice are you having that you are not getting answers to?getting answers to?getting answers to?getting answers to?
2.2.2.2. What patient problems or patient challenges do you feel What patient problems or patient challenges do you feel What patient problems or patient challenges do you feel What patient problems or patient challenges do you feel that youthat youthat youthat you’’’’re not able to address appropriately or to your re not able to address appropriately or to your re not able to address appropriately or to your re not able to address appropriately or to your satisfaction?satisfaction?satisfaction?satisfaction?
3.3.3.3. What patient problems are your patients saying back to What patient problems are your patients saying back to What patient problems are your patients saying back to What patient problems are your patients saying back to you needs more attention or more followyou needs more attention or more followyou needs more attention or more followyou needs more attention or more follow----up?up?up?up?
http://education.accme.org/video/accme-video-faq/traditional-needs-assessment
RSS evaluation goals
�Provide feedback to faculty
�Assess changes in competency
� Identify practice gaps
? Increase response rate
Final assessment tools
• Hypothesis: Less evaluation = more response rate
• Quarterly, web-based evaluation for “single discipline”RSS– Evaluates perceptions of CME quality (Shewchuk et al tool)
– Includes intent to change and needs assessment questions– SurveyMonkey
• Post-activity paper-based evaluation for “multidisciplinary” RSS– Evaluates quality of CME presentation (Wood et al tool)– Includes intent to change and needs assessment questions
Webinar pre-survey question: Have you ever used a web-based survey to evaluate CME outcomes (n = 52)
Yes, 62%
No, 38%
Table 1. Participation in case-based RSS between Apr-Jun 2009 (Q2)
RSS title Unique participants
Total sessions
Attendance average
Valid emails
Previous SurveyMonkey
opt-outs
Evaluation response rate (after 1 invite)
Cardiac Teaching Conference
41 12 20 39 1 56% (22/39)
Tumor Board –AIDHC
35 13 16 34 0 65% (22/34)
Oncology Teaching Conference
18 10 7 17 1 76% (1317)
Uroradiology Case Conference
12 9 6 10 1 80% (8/10)
Evidence-based Lecture Series
9 5 4 6 0 50% (3/6)*
Surgery M&M Case Conference
10 9 7 9 0 44% (4/9)
Patient Safety Case Conference
15 3 8 14 1 57% (8/14)**
Tumor Board- JAX 45 8 19 39 1 59% (23/39)
Surgery/Radiology Series - JAX
16 13 10 14 1 71% (10/14)
avg = 61%
*response rate after 3 survey invitations
**response rate after 2 survey invitations
Webinar pre survey: Average paper survey response rate? (n = 41)
0
10
20
30
40
50
60
70
80
90
100
# of RSS
per
cen
t
Median = 55%, range = 2 – 100!
Table 2. Response rate for quarterly, web-based RSS evaluation (2009)
RSS title Q3 Q4
Cardiac Teaching Conference 57% (27/47)* 43% (15/35)
Tumor Board – AIDHC 65% (28/43)* 52% (15/29)
Oncology Teaching Conference 73% (19/26)* 61% (11/18)*
Uroradiology Case Conference 50% (8/16) 73% (8/11)*
Evidence-based Lecture Series 80% (8/10)* 50% (2/4)*
Surgery M&M Case Conference 60% (6/10)* 50% (4/8)
Patient Safety Case Conference 40% (6/15) 43% (3/7)
Tumor Board- JAX 67% (35/52)* 59% (22/37)
Surgery/Radiology Series - JAX 82% (14/17)* 64% (9/14)*
avg = 64% avg = 55%
*response rate after 2 survey invitations
Survey example (web)
Web-based RSS email invite
Subject:Subject:Subject:Subject: CME/CE evaluation for Nemours Tumor Board
Body:Body:Body:Body:Hi [CustomValue] [LastName]:
Our records indicate that you attended one or more sessions of Nemours "Tumor Board" between July andSeptember 2009. Tumor Board typically occurs every Monday at 12:30 PM. A total of 10 sessions were heldbetween July and September 2009.
In an effort to reduce the time you spend completing course evaluations (usually one at each session), thesurvey link below will direct you to a single evaluation covering this entire period.
Here is a link to the survey: https://www.surveymonkey.com/s.aspx
In order to meet the requirements of the Accreditation Council for Continuing Medical Education to provide CME credit for this activity, we must measure the educational outcomes. Please respond to this survey (it takes < 5 minutes) so that we can continue providing credit for this series.
If you have any questions about this evaluation, please contact me via phone (904) 697-3877 or email([email protected]).
Thank you for your participation!
Jason Olivieri Nemours Office of CME
Please note: If you do not wish to receive further emails from us, please click the link below, and you will beautomatically removed from our mailing list. https://www.surveymonkey.com/optout.aspx
Q&A
We welcome your questions and comments.
Please use the chat box on the left side of your screen to submit a question.
Thank you!
Please take a moment to complete the brief evaluation.
We appreciate your feedback!
Thank You!
Join us for the Alliance’s next Takeout Tuesday webinar!
Taking Education to the Healthcare Team: In-situ Simulation in Acute MI Care as a Model for Team-
focused CME
Tuesday, June 15, 20102:00 – 3:00 PM ET
www.acme-assn.org