using resident perceptions to improve educational quality and accountability conference session:01...
TRANSCRIPT
Using Resident Perceptions to Using Resident Perceptions to Improve Educational Quality and Improve Educational Quality and
Accountability Accountability Conference Session:01Conference Session:01
Alice Edler, MD, MPH, MA (Educ) – Faculty Fellow Alice Edler, MD, MPH, MA (Educ) – Faculty Fellow Nancy Piro, PhD, Program Manager/Education SpecialistNancy Piro, PhD, Program Manager/Education Specialist
Ann M. Dohn, MA – Director, GME Ann M. Dohn, MA – Director, GME Designated Institutional Official (DIO) &Designated Institutional Official (DIO) &
Bardia Behravesh, EdD, Bardia Behravesh, EdD, Program Manager/Education SpecialistProgram Manager/Education Specialist
Department of Graduate Medical EducationDepartment of Graduate Medical EducationStanford University Medical CenterStanford University Medical Center
Session ObjectivesSession Objectives
At the end of this session, you will be able At the end of this session, you will be able to…to…1.1. Understand how we tap into trainees’ current Understand how we tap into trainees’ current
perception of the quality of GME at Stanfordperception of the quality of GME at Stanford
2.2. Understand resident perceptions of factors Understand resident perceptions of factors which influence the quality of their which influence the quality of their postgraduate medical educationpostgraduate medical education
3.3. Operationalize the use of survey data for Operationalize the use of survey data for program quality improvementprogram quality improvement
4.4. Be introduced to the use of factor analysis to Be introduced to the use of factor analysis to evaluate survey dataevaluate survey data
Annual GME HouseStaff SurveyAnnual GME HouseStaff Survey“Basics”“Basics”
Annual survey to all HouseStaff
Designed to capture HouseStaff perceptions
Questions are primarily linked to Common Program Requirements
Goal: Early warning and continual program improvement efforts
Setting the StageSetting the Stage
82 Programs (and 82 Program Directors)82 Programs (and 82 Program Directors)
1100 Trainees1100 Trainees
5 Major Affiliate Training Sites5 Major Affiliate Training Sites
Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey
Do we really want to know what they think Do we really want to know what they think of us?of us?
Why do we survey? We already have the Why do we survey? We already have the ACGME survey and annual program ACGME survey and annual program evaluations.evaluations.
What kind of questions do we ask?What kind of questions do we ask?
What do we do with the responses?What do we do with the responses?
Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey
Do we really want to know what they think Do we really want to know what they think of us?of us?
YES!!!YES!!!
Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey
Why do we survey?Why do we survey?– Want residents to be able to speak freely and Want residents to be able to speak freely and
confidentially confidentially – Want to know what is going on across multiple Want to know what is going on across multiple
geographic locations and diverse programsgeographic locations and diverse programs– Surveys- instant and easy access (more Surveys- instant and easy access (more
efficient than Town Halls)efficient than Town Halls)– Conserves Residents’ and GME Staff timeConserves Residents’ and GME Staff time– Tap into perceptions not queried in program Tap into perceptions not queried in program
evaluations or the ACGME surveyevaluations or the ACGME survey
Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey
What kind of questions do we ask?What kind of questions do we ask?– Information about the ResidentsInformation about the Residents– Overall ExperienceOverall Experience– Program ExperienceProgram Experience– Program FacultyProgram Faculty– Evaluation and Feedback Evaluation and Feedback – Training EnvironmentTraining Environment– Personal WellnessPersonal Wellness– Quality ImprovementQuality Improvement– Communication and Patient PerceptionsCommunication and Patient Perceptions
Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey
How many questions?How many questions?
Quantitative and Open Ended / CommentsQuantitative and Open Ended / Comments
How do we scale responses?How do we scale responses?
How do we maintain confidentiality?How do we maintain confidentiality?
How do we encourage residents to How do we encourage residents to respond?respond?
What do we do with the responses?What do we do with the responses?– Explore, analyze, and compare the data Explore, analyze, and compare the data
(trend by year, program vs institution).(trend by year, program vs institution).– Prepare summary reports for each programPrepare summary reports for each program
Upload for programs to use the aggregated data Upload for programs to use the aggregated data for their Annual Program Reviews (APRs)for their Annual Program Reviews (APRs)
Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey
Sample Summary ReportSample Summary Report
Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey
What do we do with the responses?What do we do with the responses?– Quality ImprovementQuality Improvement– Organizational and Programmatic ChangeOrganizational and Programmatic Change– Data for “Report Cards”Data for “Report Cards”– Early Warning SystemEarly Warning System– Chart ProgressChart Progress
Lessons LearnedLessons Learned
““Charming”Charming”
Non-digital HVAC units (cold residents)Non-digital HVAC units (cold residents)
5 Year Accreditation Cycle vs Extremely 5 Year Accreditation Cycle vs Extremely Unhappy ResidentsUnhappy Residents
Data Mining Data Mining
Exploring the Data for New Ideas…Exploring the Data for New Ideas…– Can residents tell us what elements of Can residents tell us what elements of
education are important to them?education are important to them?
Education Quality Education Quality
Teaching
Curriculum
Educational Leadership
Theories of Educational Quality Theories of Educational Quality
SkeffSkeff
Harris Harris
Cooke, Irby Cooke, Irby
O’Sullivan, Edler O’Sullivan, Edler
ACGME ACGME
? The Post Graduate Learner ?? The Post Graduate Learner ?
Why is the Adult Learner Why is the Adult Learner Important? Important?
All learners are importantAll learners are important– They are “the starting point, They are “the starting point,
the center and the end” of educational the center and the end” of educational activities (Dewey, 1902)activities (Dewey, 1902)
Adults have a lifetime of past educational Adults have a lifetime of past educational experiencesexperiences
Adults are active participants in their own Adults are active participants in their own teaching and learningteaching and learning
Adults are intrinsically motivated Adults are intrinsically motivated
Purpose of Our InvestigationPurpose of Our Investigation
To identify elements of the educational To identify elements of the educational milieu that adult learners, in this case, post milieu that adult learners, in this case, post graduate medical trainees, believe to graduate medical trainees, believe to contribute significantly to the quality of contribute significantly to the quality of their educational experiences.their educational experiences.
Overall Goals Overall Goals
To take these perceptions into account to To take these perceptions into account to help increase ownership of learning, help increase ownership of learning, promote stronger faculty/trainee promote stronger faculty/trainee relationships, and guide educational relationships, and guide educational improvement. improvement.
Methods Methods
Survey model, with questions from:Survey model, with questions from:– Literature reviews Literature reviews – Institutionally relevant themes and accreditation Institutionally relevant themes and accreditation
topics topics
Residents not fellows Residents not fellows 18 Core Residencies 18 Core Residencies AnonymousAnonymousFactor Analysis with Principal Component Factor Analysis with Principal Component AnalysisAnalysis
Core Programs StudiedCore Programs Studied
Anesthesia Anesthesia DermatologyDermatologyEmergency MedicineEmergency MedicineInternal MedicineInternal MedicineNeurosurgeryNeurosurgeryObstetrics and Obstetrics and GynecologyGynecologyOphthalmologyOphthalmologyOrthopedic Surgery Orthopedic Surgery OtolaryngologyOtolaryngology
PathologyPathologyPediatricsPediatricsPhysical Medicine & Physical Medicine & RehabRehabPlastic SurgeryPlastic SurgeryPsychiatry Psychiatry Radiation OncologyRadiation OncologyRadiology Radiology Surgery (General) Surgery (General) UrologyUrology
32 educationally oriented questions 32 educationally oriented questions
were analyzed using :were analyzed using : Bartlett's Test for adequacy of partial Bartlett's Test for adequacy of partial correlations correlations
Determination of Eigen values Determination of Eigen values – All items with Eigen values >0.5 were entered All items with Eigen values >0.5 were entered
into the factor analysis and then rotated via a into the factor analysis and then rotated via a Varimax rotation.Varimax rotation.
Response Rates and Response Rates and Demographic Data Results Demographic Data Results
88.1% Response Rate ( n = 404)88.1% Response Rate ( n = 404)M:F Ratio = 46% : 44%M:F Ratio = 46% : 44%Missing data 13% Missing data 13% PGYPGY– 1 1 17%17%– 22 19%19%– 33 17%17%– 44 19%19%– 55 13%13%– 66 6%6%
Statistical Results Statistical Results
Significant Bartlett's Test Significant Bartlett's Test
15 items with Eigen values >0.515 items with Eigen values >0.5
All factors positively correlated All factors positively correlated
3 principal components identified on Spree 3 principal components identified on Spree plot with 57% of the variance explained plot with 57% of the variance explained
αα = 0.889 = 0.889
Results Results
The spree plot suggested the presence of The spree plot suggested the presence of three principal factors. three principal factors.
Fifty-seven percent of the variance was Fifty-seven percent of the variance was explained by the three factors in this explained by the three factors in this model. model.
Spree Plot Spree Plot
Results of the Factor AnalysisResults of the Factor Analysis
Three factors Three factors emerged from the initial emerged from the initial principal component analysis. principal component analysis. – Extraction was completed at ten iterations.Extraction was completed at ten iterations.– The resultant factor loadings suggested factor The resultant factor loadings suggested factor
one describing one describing curriculum qualitycurriculum quality, factor two , factor two describing describing teaching quality teaching quality and factor three and factor three describing describing quality in educational leadershipquality in educational leadership. .
Rotated Component Matrix
Component
1 2 3 4 5
1. In my program I have received sufficient education in responding to cultural differences with patients and colleagues 0.836 0.104 0.143 0.118 0.005
2.In my program I have received sufficient education in applying ethical principles to the practice of medicine 0.809 0.111 0.25 0.02 0.114
3. In my program I have received sufficient education in communicating with non English speaking patients and their families 0.772 0.148 0.022 0.031 0.007
4.In my program I have received sufficient education in improving my communication skills 0.771 0.14 0.059 0.116 0.006
5. In my program I have received sufficient education in improving my teaching skills 0.744 0.14 0.261 0.049 0.071
6. In my program I have received sufficient education in reducing costs while working in the hospital/lab 0.645 0.292 0.041 0.17 0.009
7.What % of your faculty are effective teachers 0.189 0.791 0.212 0.025 0.074
8.How many of your faculty show a strong interest in your education 0.154 0.772 0.248 0.125 0.055
9. What % of your faculty regularly attend journal club 0.15 0.764 0.067 0.138 0.022
10. How many of your faculty regularly participate in discussions and conferences 0.168 0.759 0.138 0.119 0.117
11. What % of your faculty spend an adequate amount of time supervising residents 0.179 0.739 0.116 0.007 0.078
12. My program director considers my opinion when making decisions 0.12 0.053 0.776 0.002 0.08
13. In my program we are encouraged to ask questions 0.303 0.255 0.65 0.085 0.085
14. My program is organized to meet my educational needs 0.222 0.346 0.58 0.139 0.139
15. In my program I am treated with respect 0.025 0.395 0.58 0.089 0.089
Component 1 1. In my program I have received sufficient education in responding to cultural differences with patients and colleagues 0.836 0.104 0.143 0.118 0.005
2.In my program I have received sufficient education in applying ethical principles to the practice of medicine 0.809 0.111 0.25 0.02 0.114
3. In my program I have received sufficient education in communicating with non English speaking patients and their families 0.772 0.148 0.022 0.031 0.007
4.In my program I have received sufficient education in improving my communication skills 0.771 0.14 0.059 0.116 0.006
5. In my program I have received sufficient education in improving my teaching skills 0.744 0.14 0.261 0.049 0.071
6. In my program I have received sufficient education in reducing costs while working in the hospital/lab 0.645 0.292 0.041 0.17 0.009
Component 2
7.What % of your faculty are effective teachers 0.189 0.791 0.212 0.025 0.074
8.How many of your faculty show a strong interest in your education 0.154 0.772 0.248 0.125 0.055
9. What % of your faculty regularly attend journal club 0.15 0.764 0.067 0.138 0.022
10. How many of your faculty regularly participate in discussions and conferences 0.168 0.759 0.138 0.119 0.117
11. What % of your faculty spend an adequate amount of time supervising residents 0.179 0.739 0.116 0.007 0.078
Component 3
12. My program director considers my opinion when making decisions 0.12 0.053 0.776 0.002 0.08
13. In my program we are encouraged to ask questions 0.303 0.255 0.65 0.085 0.085
14. My program is organized to meet my educational needs 0.222 0.346 0.58 0.139 0.139
15. In my program I am treated with respect 0.025 0.395 0.58 0.089 0.089
Summary of Findings Summary of Findings
Postgraduate medical trainees can identify Postgraduate medical trainees can identify 3 key areas of educational quality.3 key areas of educational quality.
These can be understood as the These can be understood as the constructs of constructs of curriculum, teaching, and curriculum, teaching, and educational leadershipeducational leadership..
Our construct findings were validated with Our construct findings were validated with answers to questions on the ACGME answers to questions on the ACGME survey.survey.
Curriculum ConstructCurriculum Construct
Our findings validated the students’ Our findings validated the students’ understanding of curriculumunderstanding of curriculum– One facet among the highest rated was One facet among the highest rated was
medical ethics instruction medical ethics instruction We had just that year added a mandatory on line We had just that year added a mandatory on line module for medical ethics education to overcome module for medical ethics education to overcome some departmental insufficiencies in this area. some departmental insufficiencies in this area.
– The lowest rated facet was Systems-Based The lowest rated facet was Systems-Based Practice (SBP) Practice (SBP)
This was also a low ranking result in the ACGME This was also a low ranking result in the ACGME survey and our own program directors’ needs survey and our own program directors’ needs assessment assessment
So What?So What?
It is clear that residents can identify It is clear that residents can identify instructional elements and rate their instructional elements and rate their importance and inclusion in their importance and inclusion in their curriculum.curriculum.
We have also demonstrated that learners We have also demonstrated that learners who are reared in a competency based who are reared in a competency based era, regard these competencies as critical era, regard these competencies as critical to their curriculum. to their curriculum.
Teaching Construct Teaching Construct
Our residents identified 3 elements, Our residents identified 3 elements, teaching time both clinically and teaching time both clinically and didactically and, instructor enthusiasm in didactically and, instructor enthusiasm in teaching . teaching .
We believe this demonstrated that the We believe this demonstrated that the teacher and the learner are collaborators teacher and the learner are collaborators in the educational event.in the educational event.
Teaching Construct Teaching Construct
Irby, in his recommendations for medical Irby, in his recommendations for medical teaching in the post Flexner era, teaching in the post Flexner era, suggested we need to suggested we need to “promote “promote relationships with faculty who relationships with faculty who simultaneously support learners…to simultaneously support learners…to create collaborative learning create collaborative learning environments” environments” **
* Irby, DM, Cooke M, Obrien B, Calls for Reform of Medical Education by the Carnegie Foundation for the Advancement of Teaching: 1910-2010
How does this compare How does this compare with the literature?with the literature?
Qualitative studies by educational Qualitative studies by educational researchers list rapport and communication researchers list rapport and communication as the key to excellent teaching. Our findings as the key to excellent teaching. Our findings include the presence of the teacher at formal include the presence of the teacher at formal didactic and conference discussions as a didactic and conference discussions as a portion of this key element.portion of this key element.– Perhaps these formal presentations provide the Perhaps these formal presentations provide the
learner, especially the novice learner, with time to learner, especially the novice learner, with time to digest and reflect on the materials better than digest and reflect on the materials better than “on the fly” teaching during rounds. “on the fly” teaching during rounds.
Educational Leadership Construct
Emphasized – Joint decision making in education– Respectful atmosphere
How does this compare How does this compare with the literature?with the literature?
Our findings, like that of Irby et al. suggested Our findings, like that of Irby et al. suggested that the apprenticeship model of medical that the apprenticeship model of medical teaching need not be abandoned but teaching need not be abandoned but improved with curricular, instructional and improved with curricular, instructional and leadership enhancements. leadership enhancements. Quality GME education is a complex multi-Quality GME education is a complex multi-dimensional endeavor. dimensional endeavor. – Not only must curriculum and instructional Not only must curriculum and instructional
methods be taken into account, but also the methods be taken into account, but also the guidance and quality of leadership. guidance and quality of leadership.
– GME adds an additional layer of complexity.GME adds an additional layer of complexity.
In the professions teacher and learners In the professions teacher and learners must come together over the three must come together over the three
elements: curriculum, instruction and elements: curriculum, instruction and leadership, but an added unique fourth leadership, but an added unique fourth
consideration is omnipresent, consideration is omnipresent, the patientthe patient. .
Overall Conclusions Overall Conclusions
Survey data can be extremely valuable if Survey data can be extremely valuable if you are willing to share the data with your you are willing to share the data with your stakeholders – in a user friendly waystakeholders – in a user friendly way
Be prepared for some surprisesBe prepared for some surprises
Be prepared to act on your findings – Be prepared to act on your findings – develop action plansdevelop action plans
Don’t be afraid to think outside the box Don’t be afraid to think outside the box
Specific Conclusions Specific Conclusions
Quality education is complex and Quality education is complex and multidimensional. multidimensional.
Curriculum and instruction are only 2 key Curriculum and instruction are only 2 key elements, the third is educational leadership.elements, the third is educational leadership.
Our findings have corroborated with findings Our findings have corroborated with findings in the general educational literature. in the general educational literature.
Professional education adds to the Professional education adds to the complexity by a unique, omnipresent 4th complexity by a unique, omnipresent 4th layer: layer: The physician-patient relationThe physician-patient relation..