image guided interventions: rigid and deformable image ... · model 2 result. transformed by. model...
TRANSCRIPT
![Page 1: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/1.jpg)
Image guided interventions:rigid and deformable image registration, fusion,
visualization, and augmented realityDave Hawkes
University College London
[email protected]/cmic
![Page 2: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/2.jpg)
Outline of Talk
• Role of Image Registration in IGI• Dealing with Deforming and Moving
Tissues – Development of non-rigid registration
algorithms• Fast free form deformation• Biomechanical models• A combination of the two
– Dimension Reduction• Local Rigid Body Approximation• Motion Modelling• Shape Modelling• A combination of the two
![Page 3: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/3.jpg)
Image R
egistrationPlanning the Intervention:
Atlas generation, segmentation propagation, planning +
(Multimodal) pre-procedure imaging= Interventional plan
During the Intervention (Real-time):Alignment of plan to patient coordinates +
Intraoperative sensing to track tissue deformation and motion= Visualization to provide guidance in real-time
Assessment of the Intervention:Post procedure imaging and monitoring +
Comparison with pre-procedure plan
![Page 4: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/4.jpg)
MAGI system in the Operating Room:
Overlay of 3D preoperative image dataon stereo field of view of binocular operating microscope
(Edwards et al IEEE-Trans Med Imag 2000)
![Page 5: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/5.jpg)
Compensation for Deformation During Surgery
- Local Rigid Body Approximation
![Page 6: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/6.jpg)
Intraoperative 3D Power Doppler US
Registration of 3D Plan with 2D Intraoperative Imaging
Dean Barratt, Neil Kitchen and Dave Hawkes, UCL and Queen Square, London
1. Marker-based Registration
2. Updated US-based Registration
![Page 7: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/7.jpg)
Fully automated non-rigid registration based on FFDs defined by cubic B-splines:DCE-MR MammographyRueckert et al IEEE-Trans Med Imag, 1999
Difference of aligned images
Difference of original images
![Page 8: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/8.jpg)
From Free-Form Deformation (FFD) to Fast Free-Form Deformation (F3D)
Marc Modat, …, Seb Ourselin, Computer Methods and Programs in Biomedicine, 2009 (in press)
• Update all control points for each resampling
• Parzen Window estimation of Joint Histogram
• Convolution of gradient field of the cost function (NMI)
• Conjugate Gradient Optimisation
• 10 fold speed up from CPU to GPU implementation
![Page 9: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/9.jpg)
LungTarget – 4DCT end-inhale phase
(lungs segmented, images windowed to highlight internal lung structures)Jamie McClelland and Marc Modat
![Page 10: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/10.jpg)
LungSource – 4DCT end-exhale phase
![Page 11: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/11.jpg)
LungTarget – FFD registration result
(l t d i i d d t hi hli ht i t l
![Page 12: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/12.jpg)
Compensation for Deformation in Image Guided Surgery Using Biomechanical Modelling
Finite Element Modelling
• Hexahedral mesh• Finite strain formulation (large displacements)• Neo-Hookean constitutive relationship• At chest wall only allow tangential displacements• Ansys software
(Carter et al MIAR, 2006, MICCAI 2008)
![Page 13: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/13.jpg)
Registration of Prone Pre-op to Supine MR Image:(Carter et al MICCAI, 2008)
Prone target
Rigid + Fluid
Supine source
Rigid + FEM
Rigid + FEM + Fluid
Rigid only:20mm (TRE RMS)
Rigid + FEM + Fluid TRE:3mm (TRE RMS)
Guidance for breast surgery
![Page 14: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/14.jpg)
Guidance for breast surgery
3D surface from stereo photogrammetry
3D FEM
Tim Carter, Nick Beechy-Newman (Guy’s Hospital) and Dave Hawkes
![Page 15: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/15.jpg)
Image-Guided Breast Surgery
1 Stereo camera used to acquire 3D surface of breast
4 Validated using tracked ultrasound
2 Biomechanical model used to calculate displacement of cancer between prone DCE-MRI and surgery
3 Position of cancer displayed intraoperatively
(Carter et al MICCAI 2008)
![Page 16: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/16.jpg)
Compensation for Deformation and Motion in Image Directed Ablative Therapies
“Real-time” update of plan
• Radiofrequency Ablation• Photodynamic Therapy• High Intensity Focussed Ultrasound• Targetted Radiotherapy• …..
• Efficient implementations• Fast processors (e.g. GPUs)• Dimension Reduction
![Page 17: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/17.jpg)
Image Targetted Radiotherapy in the Lung
Transformed byModel 1 resultTransformed byModel 2 resultTransformed byModel 3 resultTransformed byModel 4 result
Reference Volume
McClelland, Tarte, Blackwell and Hawkes, UCL Webb and Brada, Institute of Cancer Research
Landau and Hughes, Guy’s and St. Thomas
McClelland et al Medical Physics 2006
![Page 18: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/18.jpg)
Constructing the Motion Models– Temporal Fitting
x y
z
![Page 19: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/19.jpg)
Motion modelling in lung radiotherapy
Blackall et al WCOMP 2003, McClelland et al SPIE Med Imag 2005, ESTRO 2005, Medical Physics 2006
Model error 1.7mm (RMS), slice thickness 1.5mm (McClelland et al AAPM 2008)
![Page 20: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/20.jpg)
Visually Assessing Motion Models Session 1 vs Session 2 – Patient 1
![Page 21: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/21.jpg)
Visually Assessing Motion Models Session 1 vs Session 2 – Patient 5
![Page 22: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/22.jpg)
Breathing patterns: ‘DRRs’ of THRIVE data
normal breathing deep breathing
![Page 23: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/23.jpg)
Motion compensation during reconstruction
Older liver data, working on volunteer study data
Odille et al., 2008MRM, 59:1401-1411
White et al, 2009MRM in press
motion compensated breath holdno correction
![Page 24: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/24.jpg)
Verification of Motion Model
+ Predicted position using model
O Measured position using registration
White et al ISMRM 2007Coollens et al ICCR 2007
![Page 25: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/25.jpg)
Shape modelling
![Page 26: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/26.jpg)
Using shape model to replace CT scanChan et al SPIE 2003, MICCAI 2004, Barratt et al MedIA 2008
Template Training set
3D Tracked Ultrasound Probe
Complete 3D model instantiated from a series of ultrasound images
![Page 27: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/27.jpg)
Error Maps of Instantiation Against CT of Cadaver
![Page 28: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/28.jpg)
MR Guided Endoscopic Prostatectomy(Steve Thompson, Prokar Dasgupta et al 2008)
![Page 29: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/29.jpg)
Overlay of MRI on Endoscopic view
![Page 30: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/30.jpg)
( )1 , +kk IITr
( )1 , +kk JJTr
( )kk JITr ,
( )kk IJTr ,
• Filter out points with poor correspondence using criterion:δε ≤−+−=
F
JkIkJF
IkJkI )()()()( xxxx
Left image sequence
Right image sequence
Stereo tracking: new detected feature : tracked feature in consecutive direction
: tracked feature in corresponding direction: stereo pair
consecutive direction
consecutive direction
corresponding direction
point filter
Mingxing Hu et al MICCAI 2007, 2008, 2009
![Page 31: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/31.jpg)
3D point cloud ICP registration, residual 1.87mm
3D Reconstruction from endoscopy images using Structure From Motion (SFM)
Mingxing Hu et al MICCAI 2007
![Page 32: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/32.jpg)
Generating statistical deformation models from multiple FE simulations
Statistical Motion Models (SMMs):Application to focal therapy in the prostate
Yipeng Hu, Dean Barratt, Mark Emberton et al Ultrasound derived model to intra-operative ultrasound (MICCAI 2008)
![Page 33: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/33.jpg)
Image Directed Partial Prostate Ablation• Cancer accurately targeted and critical structures
avoided.
TRE 5 patients, 48 landmarks
1.8mm (RMS) +/- 0.7mm
SMM built from 100’s of FEM examples• Pelvic anatomy• Rectal anatomy• Insufflations• Mechanical properties
![Page 34: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/34.jpg)
The Future of Interventions is Imaging
• Integration of learnt models (shape, motion and biomechanics):– Reduce dimensionality of optimisation– Constrain possible solutions– New algorithm based on combination of image similarity and biomechanics
(TLED)
• Next steps– Integration of Functional Information– Novel intra-operative Imaging and Sensing Devices for Navigation– Mechanical Devices for Improved Manipulation across Scales– Integration with Targeted Therapies
• Image Registration will remain the key enabling technology• Systems must be Validated and Validation is hard
![Page 35: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/35.jpg)
Acknowledgements• All members of CMIC and our clinical collaborators, in
particular: – Mike Gleeson, Neil Kitchen, Mark Emberton, Hash Ahmed, Steve
Halligan (UCLH); – Nick Beechy-Newman, David Landau (GSTT); – Steve Webb, Mike Brada, Gail ter Haar, Martin Leach (ICH)
• Funding support– EPSRC, MRC, CRUK, TSB-DTI, DoH, Philips, Brainlab, VisionRT,
Carestream, Siemens
![Page 36: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/36.jpg)
Thank you
![Page 37: Image guided interventions: rigid and deformable image ... · Model 2 result. Transformed by. Model 3 result. Transformed by. Model 4 result. Reference Volume. McClelland, Tarte,](https://reader036.vdocuments.site/reader036/viewer/2022071017/5fd0a831b289d413a737e3b0/html5/thumbnails/37.jpg)
An Excellent Book:
Image Guided InterventionsEd: Terry Peters and Kevin Cleary
Publ: Springer 2008
An Excellent Meeting: