automatic detection and segmentation of robot-assisted surgical motions presented by henry c. lin...
TRANSCRIPT
Automatic Detection and Segmentation of Robot-Assisted Surgical Motions
presented by
Henry C. Lin
Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy,
Dr. David D. Yuh, Dr. Allison M. Okamura,
Dr Gregory D. Hager
2MICCAI 2005
ERC-CISST Johns Hopkins University
AuthorsHenry LinPhD Student
Computer Science
Izhak ShafranResearch Scientist
ECE
Todd MurphyMS, 2004
Mechanical Engineering
David YuhSurgeon
Cardiac Surgery
Allison OkamuraAssistant Professor
Mechanical Engineering
Gregory HagerProfessor
Computer Science
3MICCAI 2005
ERC-CISST Johns Hopkins University
Can we automatically detect and segment the surgical motions common in both videos?
Can we quantitatively and objectively determine which video is an expert surgeon and which is an intermediate surgeon?
Motivation
QuickTime™ and aCinepak decompressor
are needed to see this picture.
QuickTime™ and aCinepak decompressor
are needed to see this picture.
QuickTime™ and aCinepak decompressor
are needed to see this picture.
QuickTime™ and aCinepak decompressor
are needed to see this picture.
Expert Intermediate
4MICCAI 2005
ERC-CISST Johns Hopkins University
Cartesian Position Plots - Left Manipulator
--- Pull suture with left hand
--- Move to middle with needle
Expert Surgeon - trial 4 Intermediate Surgeon - trial 22
5MICCAI 2005
ERC-CISST Johns Hopkins University
Previous Work
• Darzi, et al. Imperial College Surgical Assessment Device
(ICSAD) quantified motion information by tracking electromagnetic markers on a trainee’s hands.
• Rosen, et al.Used force/torque data from laparoscopic
trainers to create a hidden Markov model task decomposition specific to each surgeon.
6MICCAI 2005
ERC-CISST Johns Hopkins University
Goals
• Train LDA-based statistical models with labeled motion data of an expert surgeon and an intermediate surgeon.
• Be able to accurately parse unlabeled raw motion data into a labeled sequence of surgical gestures in an automatic and efficient way.
• Ultimately create evaluation metrics to benchmark surgical skill.
7MICCAI 2005
ERC-CISST Johns Hopkins University
Corpus
78 motion variables acquired at 10Hz(we use 72 of them)
4-throw suturing task
15 expert trials12 intermediate trials
each trial roughly 60 seconds in length
8MICCAI 2005
ERC-CISST Johns Hopkins University
Gesture Vocabulary1. Reach for needle
2. Position needle
3. Insert and push needle through tissue
6. Pull suture with left hand
4. Move to middle with needle(left hand)
8. Orient needle with both hands
7. Pull suture with right hand
5. Move to middle with needle(right hand)
9MICCAI 2005
ERC-CISST Johns Hopkins University
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
System Approach
APIsignals
X(1,t)
X(72,t)
X(t)Local
FeatureExtraction
10MICCAI 2005
ERC-CISST Johns Hopkins University
Local Feature Extractions
X(kt)
+ + + +
X(kt-m+1) X(kt+m-1)
+ +
X(kt-m) X(kt+m)
€
L(kt ) = [X(kt−m ) | X(kt−m+1) | ... | X(kt ) | ... | X(kt+m−1) | X(kt+m )]
|L(kt)| = (2m+1)|X(kt)| Example: m=5, |L| = 792
11MICCAI 2005
ERC-CISST Johns Hopkins University
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
System Approach
APIsignals
X(1,t)
X(72,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
12MICCAI 2005
ERC-CISST Johns Hopkins University
Linear Discriminant Analysis
x1
x2
The objective of LDA is to perform dimensionality reduction while preserving as much of the class discriminatory information as possible.
13MICCAI 2005
ERC-CISST Johns Hopkins University
where the linear transformation matrix W is estimated by maximizing the Fisher discriminant.
Linear Discriminant Analysis
€
Y (k) =W (N(k))
Fisher discriminant - ratio of distance between the classes and the average variance of each class
LDA
class-labeledmotion data
expected reducedoutput dimension
reduceddimension
motion data
14MICCAI 2005
ERC-CISST Johns Hopkins University
LDA Reduction (6 Labeled Classes, 3 Dimensions)Expert Surgeon
15MICCAI 2005
ERC-CISST Johns Hopkins University
LDA Reduction (6 Labeled Classes, 3 Dimensions)Intermediate Surgeon
16MICCAI 2005
ERC-CISST Johns Hopkins University
Storage Savings of LDA
Method Temporal Neighbors (m) Space required (values)
Raw no - 432,000
Raw + temporal
yes 5 4,752,000
LDA-based yes 5 18,000
For a 10 minute procedure (6000 input samples)
17MICCAI 2005
ERC-CISST Johns Hopkins University
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
System Approach
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
18MICCAI 2005
ERC-CISST Johns Hopkins University
Results
• ‘Leave 2 out’ cross-validation paradigm used. 15 expert trials, 15 rounds
• Output of 2 test trials were compared against the manually labeled data.
• The average across the 15 tests was used to measure performance.
Training set (13) Test set (2)
19MICCAI 2005
ERC-CISST Johns Hopkins University
Results
20MICCAI 2005
ERC-CISST Johns Hopkins University
Resultsn Number of
labeled classesLDA output dimensions
% correct
1 6 3 91.26
2 6 4 91.46
3 6 5 91.14
4 5 3 91.06
5 5 4 91.34
6 5 3 92.09
7 5 4 91.92
8 4 3 91.88
21MICCAI 2005
ERC-CISST Johns Hopkins University
Contributions• An automated and space efficient method to
accurately parse raw motion-data into a labeled sequence of surgical motions.
• Results support previous work that there exist quantitative differences in the varying skill levels of surgeons.
• Linear discriminant analysis is a useful tool for separating surgical motions.
22MICCAI 2005
ERC-CISST Johns Hopkins University
Future Work
• Currently getting synchronized stereo video and API data. Will allow vision-based segmentation methods to complement our statistical methods.
• Apply to a larger set of expert surgeons to other representative surgical tasks.
• Create performance metrics to be used as benchmarks for surgical skill evaluation.
23MICCAI 2005
ERC-CISST Johns Hopkins University
Acknowledgements
• Minimally Invasive Surgical Training Center at the Johns Hopkins Medical School (MISTC-JHU)
- Dr. Randy Brown, Sue Eller
• Intuitive Surgical Inc.- Chris Hasser, Rajesh Kumar
• National Science Foundation
Automatic Detection and Segmentation of Robot-Assisted Surgical Motions
Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy,
Dr. David D. Yuh, Dr. Allison M. Okamura,
Dr Gregory D. Hager
Thank you! Any questions?