a novel method and system for stereotactic …...stefan weber. "augmented reality image overlay...
TRANSCRIPT
A Novel Method and System for Stereotactic Surgical Procedures
Guang Yang1,2,3, Haidong Huang1,2, Boyang Wang1,2, Cheng Wen1,2,Yingsong Huang1,2, Yifan Fu1,2,
Yu Su1,2, Jian Wu1,2 1Department of Biomedical Engineering, Tsinghua University, Beijing 100084, China
2Graduate School at Shenzhen, Tsinghua University, Shenzhen 518055, China 3Weldon School of Biomedical Engineering, Purdue University, West Lafayette, IN 47907, United States
With the development of digital imaging technology, image-guided surgery (IGS) or surgical navigation
has become one of the most rapidly developed techniques in minimally invasive surgery (MIS) in the past
twenty years [1-11]. In conventional surgical navigation, the display used for the surgical navigation system
is often placed in a non-sterile field away from the surgeon. This forces the surgeon to take extra steps to
match guidance information on the display with the actual anatomy of the patient. This hand-eye
coordination problem has been a big challenge. Recently, augmented reality (AR) technologies have been
widely employed in IGS [12], e.g. head-mounted display (HMD) [13-14]. This system still has the problem
of motion parallax lag and lacks multi-observers’ field of vision. Another AR technology called image
overlay [15-17] is emerging rapidly. With image overlay, computer generated anatomical models of lesion
areas or the 3D structures reconstructed from medical images (computer tomography/magnetic resonance
imaging, CT/MRI) are projected onto the patients’ skin and registered with actual lesion areas precisely.
Even this technology suffers from serious shortcomings. First, the direct projection onto the skin surface
cannot produce a 3D image. Second, uneven skin surface leads to large distortions. Third, projected images
do not match actual lesion areas. Additionally, these systems still suffer the problems of a lag for motion
parallax, lack of natural view for multiple observers and visual fatigue [18]. The semi-transparent mirror
technology has also been used in surgical navigation [19-21], but it is complex to operate.
We present here a novel AR technology based on passive polarized stereo projection and two semi-
transparent mirrors for surgical navigation. The advantage of this technology is that the 3D images are
created by computer reconstruction based on the 3D voxel data collected from the medical imaging
equipment (multi-slice computed tomography scanner, MSCT) and the polarized stereo projection system. This
provides geometrically accurate 3D spatial images and reproduces motion parallax with polarized eye
glasses.
A set of accurate spatial image registration methods were developed for registration of 3D virtual images
and the corresponding lesion areas. These methods are based on the co-ordinate transformation relationship
of the world co-ordinate system, the CT image co-ordinate system, the view plane co-ordinate system of
the vtkCamera (i.e. a virtual camera for 3D rendering) and the image plane co-ordinate system of projectors.
Based on the above transformation relationship, the specific adjustment strategy includes 2 parts: size
adjustment and posture adjustment of projected images through the PC control software.
The PC control software consists of three views: the main view, the left projection view and the right
projection view. The 3D structure is reconstructed by surface rendering using the marching cubes algorithm.
From preliminary phantom registration experiments, we show that the average error of 4 selected surface
markers (<3mm) meets the clinical requirement. The trapezium distortion of the projectors is 0.6 pixel
distance. The differences of observer’s height, inter-pupillary distance and motion have little effect on the
registration accuracy. The human eye may have automatic focusing functions which increase the actual
registration accuracy.
1. Research reported in this publication was supported by the National Key Scientific Instrument and Equipment Development
Project (81427803) and the Knowledge Innovation Program of Basic Research Projects of Shenzhen Grant No.
JCY20130402145002404 and No. JCY20140408153331811.
REFERENCES
[1] Roberts, David W., John W. Strohbehn, John F. Hatch, William Murray, and Hans Kettenberger.
"A frameless stereotaxic integration of computerized tomographic imaging and the operating
microscope." Journal of neurosurgery 65, no. 4 (1986): 545-549.
[2] Barnett, Gene H., Donald W. Kormos, Charles P. Steiner, and Joe RT Weisenberger. "Use of a
frameless, armless stereotactic wand for brain tumor localization with two-dimensional and three-
dimensional neuroimaging." Neurosurgery 33, no. 4 (1993): 674-678.
[3] Reinhardt, Hans F., Gerhard A. Horstmann, and Otmar Gratzl. "Sonic stereometry in
microsurgical procedures for deep-seated brain tumors and vascular
malformations." Neurosurgery 32, no. 1 (1993): 51-57.
[4] Heilbrun, M. Peter, Paul McDonald, Clay Wiker, Spencer Koehler, and William Peters.
"Stereotactic localization and guidance using a machine vision technique." Stereotactic and
functional neurosurgery 58, no. 1-4 (1992): 94-98.
[5] Kato, Amami, Toshiki Yoshimine, Toru Hayakawa, Yoshiaki Tomita, Takuya Ikeda, Masanori
Mitomo, Koushi Harada, and Heitaro Mogami. "Computer assisted neurosurgery: development of
a frameless and armless navigation system (CNS navigator)." No shinkei geka. Neurological
surgery 19, no. 2 (1991): 137-142.
[6] Burckhardt, C. W., P. Flury, and D. Glauser. "Stereotactic brain surgery." IEEE Engineering in
Medicine and Biology Magazine 14, no. 3 (1995): 314-317.
[7] Masamune, Ken, Etsuko Kobayashi, Yoshitaka Masutani, Makoto Suzuki, Takeyoshi Dohi,
Hiroshi Iseki, and Kintomo Takakura. "Development of an MRI-compatible needle insertion
manipulator for stereotactic neurosurgery." Journal of Image Guided Surgery1, no. 4 (1995): 242-
248.
[8] Tian, Heqiang, Chenchen Wang, Xiaoqing Dang, and Lining Sun. "A 6-DOF parallel bone-
grinding robot for cervical disc replacement surgery." Medical & Biological Engineering &
Computing (2017): 1-15.
[9] Cleary, Kevin, Filip Banovac, David Lindisch, Vance Watson, and D. Stoianvici. "Robotically
assisted spine needle placement: program plan and cadaver study." In Computer-Based Medical
Systems, 2001. CBMS 2001. Proceedings. 14th IEEE Symposium on, pp. 339-342. IEEE, 2001.
[10] Guthart, Gary S., and J. Kenneth Salisbury. "The Intuitive/sup TM/telesurgery system: overview
and application." In Robotics and Automation, 2000. Proceedings. ICRA'00. IEEE International
Conference on, vol. 1, pp. 618-621. IEEE, 2000.
[11] Boehm, D. H., H. Reichenspurner, C. Detter, M. Arnold, H. Gulbins, B. Meiser, and B. Reichart.
"Clinical use of a computer-enhanced surgical robotic system for endoscopic coronary artery
bypass grafting on the beating heart." The Thoracic and cardiovascular surgeon 48, no. 04
(2000): 198-202.
[12] Okamoto, Tomoyoshi, Shinji Onda, Katsuhiko Yanaga, Naoki Suzuki, and Asaki Hattori.
"Clinical application of navigation surgery using augmented reality in the abdominal
field." Surgery today 45, no. 4 (2015): 397-406.
[13] Chen, Xiaojun, Lu Xu, Yiping Wang, Huixiang Wang, Fang Wang, Xiangsen Zeng, Qiugen
Wang, and Jan Egger. "Development of a surgical navigation system based on augmented reality
using an optical see-through head-mounted display." Journal of biomedical informatics 55
(2015): 124-131.
[14] Keller, Kurtis, Andrei State, and Henry Fuchs. "Head mounted displays for medical use." Journal
of Display Technology 4, no. 4 (2008): 468-472.
[15] Volonté, Francesco, François Pugin, Pascal Bucher, Maki Sugimoto, Osman Ratib, and Philippe
Morel. "Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic
surgery: not only a matter of fashion." Journal of hepato-biliary-pancreatic sciences 18, no. 4
(2011): 506-509.
[16] Gavaghan, Kate A., Matthias Peterhans, Thiago Oliveira-Santos, and Stefan Weber. "A portable
image overlay projection device for computer-aided open liver surgery." IEEE transactions on
biomedical engineering 58, no. 6 (2011): 1855-1864.
[17] Gavaghan, Kate Alicia, Sylvain Anderegg, Matthias Peterhans, Thiago Oliveira-Santos, and
Stefan Weber. "Augmented reality image overlay projection for image guided open liver ablation
of metastatic liver cancer." In Workshop on Augmented Environments for Computer-Assisted
Interventions, pp. 36-46. Springer, Berlin, Heidelberg, 2011.
[18] Liu, Sheng, Dewen Cheng, and Hong Hua. "An optical see-through head mounted display with
addressable focal planes." In Mixed and Augmented Reality, 2008. ISMAR 2008. 7th IEEE/ACM
International Symposium on, pp. 33-42. IEEE, 2008.
[19] Liao, Hongen, Nobuhiko Hata, Susumu Nakajima, Makoto Iwahara, Ichiro Sakuma, and
Takeyoshi Dohi. "Surgical navigation by autostereoscopic image overlay of integral
videography." IEEE Transactions on Information Technology in Biomedicine 8, no. 2 (2004):
114-121.
[20] Nakajima, Susumu, Sumihisa Orita, Ken Masamune, Ichiro Sakuma, Takeyoshi Dohi, and Kozo
Nakamura. "Surgical navigation system with intuitive three-dimensional display." In Medical
Image Computing and Computer-Assisted Intervention–MICCAI 2000, pp. 3-20. Springer
Berlin/Heidelberg, 2000.
[21] Wang, Junchen, Hideyuki Suenaga, Kazuto Hoshi, Liangjing Yang, Etsuko Kobayashi, Ichiro
Sakuma, and Hongen Liao. "Augmented reality navigation with automatic marker-free image
registration using 3-D image overlay for dental surgery." IEEE transactions on biomedical
engineering 61, no. 4 (2014): 1295-1304.
A Novel Method and System for Stereotactic Surgical Procedures
Guang Yang1,2,3, Haidong Huang1,2, Boyang Wang1,2, Cheng Wen1,2, Yingsong Huang1,2, Yifan Fu1,2, Yu Su1,2, Jian Wu1,2
1Department of Biomedical Engineering, Tsinghua University, Beijing 100084, China 2Graduate School at Shenzhen, Tsinghua University, Shenzhen 518055, China3Weldon School of Biomedical Engineering, Purdue University, West Lafayette, IN 47907, United States
Fig. 2 System configuration: instrumentation for stereotactic surgical system. Inorder to achieve the fusion of computer-generated 3D images and real objects, that is a kind of“perspective ” effect, we developed a novel system for stereotactic surgical procedures.
Abstract
Fig. 3 Principle of spatial image registration methods. These methods are based on
the co-ordinate transformation relationship between the coordinate system. W: the world
coordinate system, CT: computer tomography image coordinate system, VTK: the view plane
coordinate system of the visualization toolkit, Pr: the image plane coordinate system of
projectors.
Passive Polarized Stereo Projection Technology
Posture Adjustment Strategy of Projected Images
System Configuration
• Image-guided surgery (IGS) or surgical navigation has become one of the mostrapidly developed techniques in minimally invasive surgery (MIS) in the past 20years.
• Conventional methods have the hand-eye coordination problem. Augmentedreality (AR) technologies such as head-mounted display (HMD) has the problemof motion parallax lag and lacks multi-observers’ field of vision. Another ARtechnology called image overlay cannot produce a 3D image and leads to largedistortions.
• We demonstrate a novel AR technology based on passive polarized stereoprojection and two semi-transparent mirrors for surgical navigation.
• The advantage of this technology is that the 3D images are created by computerreconstruction based on the 3D voxel data collected from the medical imagingequipment and the polarized stereo projection system which providesgeometrically accurate 3D spatial images and reproduces motion parallax withpolarized eye glasses.
• A set of accurate spatial image registration methods were developed forregistration of 3D virtual images and the corresponding lesion areas with apreliminary phantom markers registration result of less than 3 mm which meetsthe clinical requirement.
Spatial Image Registration PC Control Software
Fig. 5 PC control software interface. The PC control software consists of three views: the
main view, the left projection view and the right projection view. Three 3Dstructures are reconstructed using surface rendering with marching cubesalgorithm. Two 3D structures in the left projection view and the right projectionview have certain angles based on the parallax of the two eyes.
Preliminary Phantom Registration Result
Fig. 6 Phantom registration result. There exist 8 surface markers in the phantom. Due to
the viewing angle, the actual markers can be observed are 4 ones, namely point mark 2, 5, 7, 8
thus registration errors analysis is based on these reference markers. The two observers’ height
selected are 1650mm and 1750mm. The interpupillary distance is selected as 65mm.
Fig. 4 Framework of posture adjustment strategy of projected images. The postureadjustment of projected images includes two parts: 1) coarse adjustment of hardwarecomponent; 2) software adjustment of physical spatial coordinates of markers in the 2D
projected images (𝑞51′′ and 𝑞52
′′ ) to approach the target positions of corresponding markers (𝑞51′
and 𝑞52′ ) through changing the corresponding positions of the points in the view plane coordinate
system based on the above transformation relationship and calculate the image registration errors
to evaluate the posture adjustment effect.
1
1
2
3D movie
Fig. 1 Passive Polarized Stereo Projector Theory. Each front of projector lens is fixedwith one linear polarized film with phase differences of 90 degrees to guarantee that thepolarization state of two 2D projected images are mutually perpendicular to each other. Semi-transparent mirror 1 has a certain degree of fog and a layer of metal film thus implementing thefunction of diffuse reflection and permeability. Semi-transparent mirror 2 can achieve thefunction of specular reflection and permeability.
Research reported in this publication was supported by the National Key Scientific Instrument and Equipment
Development Project (81427803) and the Knowledge Innovation Program of Basic Research Projects of Shenzhen
Grant No. JCY20130402145002404 and No. JCY20140408153331811.