beta test results of a new system assessing laparoscopic competence lee l. swanstrom, md, portland,...
TRANSCRIPT
![Page 1: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/1.jpg)
Beta Test Results of a New System Assessing
Laparoscopic Competence
Lee L. Swanstrom, MD, Portland, ORNathaniel J. Soper, MD, Chicago, IL
Gerald M. Fried, MD, Montreal, CanadaKaaren Hoffman, Ph.D., Los Angeles, CA
And the FLS Committee of SAGES
![Page 2: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/2.jpg)
Background Education in laparoscopic skills haphazard
and often directed to specific procedure(s)
Much education provided by industry
Underlying knowledge of physiology and hazards variable and often inadequate
No established tests of competency in laparoscopic cognitive knowledge and technical skills
![Page 3: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/3.jpg)
SAGES Fundamentals of Laparoscopic Surgery (FLS) Program
Developed 1997-2004 Designed to teach and assess cognitive knowledge
and technical skills ‘fundamental to the performance of basic laparoscopic surgery’
Cognitive material in CD-ROM ‘text’ Cognitive examination developed with USC educators
to achieve ‘high-stakes’ level Manual skills instruction and examination modified
from MISTELS (McGill U.)
![Page 4: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/4.jpg)
CD ROM Content
Disc OneI. Preoperative ConsiderationsII. Intraoperative ConsiderationsIII. Basic Laparoscopic Procedures IV. Postoperative Considerations
Disc TwoV. Manual Skills Instruction and Practice
![Page 5: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/5.jpg)
![Page 6: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/6.jpg)
Recognition of Laparoscopic Findings
Crohn’s Disease Peritoneal Mets
![Page 7: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/7.jpg)
Recognize Complications
![Page 8: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/8.jpg)
MISTELS:From Fried,et al, McGill
![Page 9: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/9.jpg)
FLS Videotrainer Box
Portable unit includes camera, light, power supply and connection for TV or computer
Includes material for MISTELS tasks
![Page 10: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/10.jpg)
Cognitive Examination
Computer based, visuals included, approximately 70 questions.
- Multiple choice questions (from pool)
- Clinical scenarios (from pool) Rigid question design to test blueprint Content validated by panels of experts Assesses cognitive knowledge and clinical
judgment
![Page 11: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/11.jpg)
Manual Skills Examination Modified from MISTELS examination
developed at McGill University
Videotrainer box-based, proctored
Extensive validation studies published over past 10 years attesting to face, construct, concurrent and predictive validity (Fried, et al, Ann Surg 240: 518-28, 2004)
Minor changes in protocol (trainer box, paring skills sets) demonstrated to be inconsequential
![Page 12: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/12.jpg)
Methods: Validation of FLS Examinations
Beta testing phase:
No studying of CD-ROM material, except brief explanation of manual skills
Administration of the FLS test at Beta sites with “known” novice & expert groups
7 centers in N. America testing PGY-2, PGY-5, MIS Fellows and attending surgeons
Self-assessment of experience and skill level
![Page 13: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/13.jpg)
Data Analyses
Analyses of Variance with A Priori Contrasts to detect performance differences across levels of Surgical Training and Laparoscopic Experience.
Regression and Path Analyses to determine the unique effect of Laparoscopic Experience upon performance.
Independent T-tests to detect differences across Self-Reported Laparoscopic Competency Levels
![Page 14: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/14.jpg)
Construct Validity
Are there performance differences across groups with different degrees of expertise in laparoscopic surgery?
![Page 15: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/15.jpg)
Cognitive Performance by Training Level (N=65)
100
90
80
70
60
50
40SurgeonFellowPGY5PGY2
Per
cen
tage
Sco
re
![Page 16: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/16.jpg)
Cognitive Performance byLaparoscopic Experience Quartiles (N=54)
100
90
80
70
60
50
404th3rd2nd1stExperience Quartiles
Per
cen
tage
Sco
re
![Page 17: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/17.jpg)
Separating General Training and Laparoscopic ExperienceHierarchical Regression of Cognitive Assessment (N=54)
Predictor
Training EnteredLap. Experience Added
Multiple R
.675
.766
R2 Change
.131
Signif.
p<.000
Correlation of Training and Assessment with Experience held constant (Part Correlation) = .127
Correlation of Experience and Assessment with Training held constant (Part Correlation) = .362
![Page 18: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/18.jpg)
Path Analysis - Direct vs (Indirect) Effects
Training
Laparoscopic Experience
Cognitive Assessment
.79
.21 (.46)
.59 (.16)
![Page 19: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/19.jpg)
Cognitive Performance bySelf-Reported Competence (N=54)
Above AverageAverage and Below
100
90
80
70
60
50
40
Per
cen
tage
Sco
res
![Page 20: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/20.jpg)
Manual Skill Performance by Training Level (N=58)
Nor
mal
ized
Sco
res
SurgeonFellowPGY5PGY2
100
80
60
40
20
0
![Page 21: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/21.jpg)
Manual Skill Performance by Laparoscopic Experience Quartiles (N=48)
4th3rd2nd1st
100
80
60
40
20
0
Experience Quartiles
Nor
mal
ized
Sco
res
![Page 22: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/22.jpg)
Manual Skill Performance bySelf-Reported Competence (N=48)
100
80
60
40
20
0
Above AverageAverage and Below
Nor
mal
ized
Sco
res
![Page 23: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/23.jpg)
Separating General Training and Laparoscopic Experience – Manual Skills
Regression and Path Analyses were not undertaken as the relationships between the Manual Skills assessment, laparoscopic experience and general training were non-linear.
Previous studies with the Manual Skills assessment have established a direct link between assessment performance and laparoscopic skills rated in laboratory and clinical situations.
![Page 24: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/24.jpg)
Validity Question # 1
Are there performance differences acrossgroups with different degrees of expertise in laparoscopic surgery?
Yes. Both the cognitive and manual skills assessments are able to discriminate between groups with different levels of laparoscopic expertise.
![Page 25: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/25.jpg)
Validity Question # 2
Are the noted performance differences specific to laparoscopic expertise as opposed to general expertise?
Yes. Increases seen with general surgical training are primarily related to increases in laparoscopic experience.
![Page 26: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/26.jpg)
FLS Beta Test: Other Findings
Appropriate psychometric properties
Viable administrative procedures
Viable standard-setting mechanism
![Page 27: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/27.jpg)
Conclusions FLS should provide a uniform curriculum for
the acquisition of basic knowledge and technical skills in laparoscopy
FLS is a reliable and valid process for testing of ‘competence’ in basic laparoscopy— passage, “…demonstrates the knowledge and skills fundamental to the performance of laparoscopic surgery”
This may become a model for training as more technology-dependent surgical advances are introduced
![Page 28: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/28.jpg)
Next Steps
ACS has recently partnered with SAGES for dissemination of the FLS program
6 Test Centers established (Tufts U., Boston; U. Penn., Philadelphia; McGill U., Montreal; Wash. U., St. Louis; Legacy Hosp., Portland; USC, Los Angeles)
Testing available at ACS Clinical Congress and at SAGES annual meeting
![Page 29: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/29.jpg)
Developed by SAGES
![Page 30: Beta Test Results of a New System Assessing Laparoscopic Competence Lee L. Swanstrom, MD, Portland, OR Nathaniel J. Soper, MD, Chicago, IL Gerald M. Fried,](https://reader037.vdocuments.site/reader037/viewer/2022110116/5518c54e550346881f8b57cf/html5/thumbnails/30.jpg)