1

A System for Video-based Navigation forEndoscopic Endonasal Skull Base Surgery

Daniel J. Mirota, Student Member, IEEE, Hanzi Wang*, Senior Member, IEEE, Russell H. Taylor, Fellow, IEEE,Masaru Ishii, Gary L. Gallia, and Gregory D. Hager, Fellow, IEEE

Abstract—Surgeries of the skull base require accuracy tosafely navigate the critical anatomy. This is particularly the casefor endoscopic endonasal skull base surgery (ESBS) where thesurgeons work within millimeters of neurovascular structures atthe skull base. Today’s navigation systems provide approximately2 mm accuracy. Accuracy is limited by the indirect relationshipof the navigation system, the image and the patient. We proposea method to directly track the position of the endoscope usingvideo data acquired from the endoscope camera. Our methodfirst tracks image feature points in the video and reconstructs theimage feature points to produce three-dimensional (3D) points,and then registers the reconstructed point cloud to a surfacesegmented from pre-operative computed tomography (CT) data.After the initial registration, the system tracks image features andmaintains the two-dimensional (2D)-3D correspondence of imagefeatures and 3D locations. These data are then used to updatethe current camera pose. We present a method for validationof our system, which achieves sub-millimeter (0.70 mm mean)target registration error (TRE) results.

Index Terms—Endoscopy, Registration, Image-guided treat-ment, Surgical guidance/navigation

I. INTRODUCTION

Endoscopic endonasal skull base surgery (ESBS) has gainedmuch interest recently over traditional open surgical approachesas a treatment for pathologies involving the skull base andparanasal sinuses. Pituitary lesions, though generally benign,are the most common skull base lesion. ESBS is commonly

Copyright (c) 2011 IEEE. Personal use of this material is permitted. However,permission to use this material for any other purposes must be obtained fromthe IEEE by sending a request to pubs-permissions@ieee.org.

Manuscript received April 11, 2011; revised September 26, 2011; acceptedNovember 3, 2011. This work was supported principally by the NationalInstitutes of Health under grant number R21 EB005201. Further supportwas provided by the NSF ERC Grant EEC9731748, the Link Fellowship inSimulation and Training, the Medtronic Computer-Aided Surgery ResearchAward and by Johns Hopkins University. Asterisk indicates correspondingauthor.

D. J. Mirota is with Department of Computer Science, Johns HopkinsUniversity, Baltimore, MD, USA (email: dan@cs.jhu.edu).

*H. Wang is with School of Information Science and Technology,Xiamen University, Xiamen, China; and Fujian Key Laboratory ofthe Brain-like Intelligent Systems (Xiamen University), Fujian, China(email: hanzi.wang@ieee.org).

R. H. Taylor is with Department of Computer Science, Johns HopkinsUniversity, Baltimore, MD, USA (email: rht@jhu.edu).

M. Ishii is with Department of Otolaryngology–Head and NeckSurgery, Johns Hopkins Medical Institutions, Baltimore, MD, USA(email: mishii3@jhmi.edu).

G. L. Gallia is with Department of Neurosurgery and Oncology, Johns Hop-kins Medical Institutions, Baltimore, MD, USA (email: ggallia1@jhmi.edu).

G. D. Hager is with Department of Computer Science, Johns HopkinsUniversity, Baltimore, MD, USA (email: hager@cs.jhu.edu).

Color versions of one or more of the gures in this paper are available onlineat http://ieeexplore.ieee.org.

used to treat pituitary lesions, non-neoplastic skull base lesionsand other tumors of the skull base and nasal cavity. Traditionalsurgical approaches to the skull base are associated with signif-icant morbidities because healthy cranial nerves are sometimesinjured during surgery. Unlike traditional approaches, ESBSis less invasive and is shown to reduce operative time anddecrease the length of hospital stay [1].

ESBS and traditional approaches are best contrasted with aclinical example. Fig. 1 shows the CT and magnetic resonanceimaging (MRI) scans of a patient with a clival chordoma. Thecentral location of this tumor makes it difficult to approachusing traditional means, especially because the tumor involvesthe clivus and extends behind the carotid artery. This tumorwas removed using an endoscopic endonasal approach. In thiscase the carotid artery was exposed via a vidian canal drill-out.The endoscopic images were taken before and just after thetumor was resected. Notice the carotid artery is completelyfree of the tumor at the end of the resection. Manipulatingsuch high-value structures in cases like this is the reason thatESBS requires precise knowledge of patient anatomy. Thus,surgical navigation is key for success, especially to aid juniorsurgeons and for complex cases [1], as it provides the surgeona means to both maintain orientation and to monitor progress.

In current practice, the surgeon uses a pointer tool to interactwith the navigation system. The system tracks rigid bodiesattached to the tool and the patient. During preparation forsurgery, the rigid body attached to the patient is registeredto fiducial markers on the patient. The rigid body definesthe patient’s head in the navigation system. The navigationsystem in turn calculates a rigid-body transformation betweenthe patient’s head and the tool to display the correspondinglocation in CT. A drawback of the current procedure is thateach rigid-body transformation measurement contributes tolocalization error. In fact, localization error is typically quotedas 2 mm with a good registration, and can be much larger witha poor one [2]. Errors of this magnitude could lead to surgicalerror resulting in high morbidity or mortality.

To improve current navigation systems, we propose a newsystem that utilizes endoscopic video data for navigation. Whileendoscopic video presents many challenges including reflection,specularity, and low texture, our system robustly handles thesechallenges and creates a 3D reconstruction from video. Thesystem then registers the 3D reconstruction to a pre-operativeCT scan. After the initial registration, the system tracks thecamera location by matching image features and performingrobust 2D-3D pose estimation. Instead of relying on the longrigid-body transformation chain that current navigation systems

2

Pre-resection Post-resectionPre-operative Post-operative

Carotid Artery

CT CT

MRI

Clivus

Tumor

Carotid Artery

Carotid Artery

Carotid Artery

Vidian Artery and Nerve

Carotid Artery

Carotid Artery

Clivus

MRI

Fig. 1: Preoperative (left column) and postoperative (right column) axial CT (upper) and MRI (lower) scans of a patient andendoscopic images (middle) illustrate how clear identification of the carotid artery (circled in the CT and MRI scans) is crucialto a successful resection of the tumor. In this case, the left carotid artery was exposed in order to remove tumor extendingposterior to the artery.

use, video-CT registration employs a more direct, accuratelocalization of the camera relative to the patient. We showsub-millimeter target registration error (TRE) results.

Tracking the location of a camera relative to CT has beenstudied in other areas of image-enhanced surgical navigation.In 2002, Shahidi et al. [3] presented a system for endoscopecalibration and image-enhanced endoscopy. They achievedmillimeter accuracy for a system using passive optical markersfor tracking. More recently, Lapeer et al. [4] evaluated a similarsystem, again using passive optical markers for tracking andreported that sub-millimeter accuracy still remains elusive. Vari-ous tracker-based methods have appeared recently, including [5]and [6], where a system similar to an Optotrak was used to trackthe endoscope and provided an augmented display. Schulze et al.[7] reported a system using the Medtronic StealthStation alongwith novel rendering techniques to provide a side-by-side liveversus virtual endoscopy. Daly et al. [8] similarly presenteda tracker-based approach using a Polaris and demonstratedtracking with intra-operative cone-beam CT.

Video registration has been previously applied to bron-choscopy [9] where normalized mutual information was usedto register to CT. In [10], visual tracking and registrationwas demonstrated on a skull phantom. The work here is anextension and generalization of [10]. Similar to our work,Allian et al. [11] presented an epipolar geometry methodused to re-localize the endoscope image at a biopsy site. Ourproblem differs in that we aim to reconstruct, register andtrack the endoscope in a rigid environment, in contrast tolocalizing a single, usually defined, target. In addition, ourwork differs from [11] by using Singular Value Decomposition(SVD) Match [12] instead of block matching and the AdaptiveScale Kernel Consensus (ASKC) [13] instead of Maximum APosteriori Sample Consensus (MAPSAC). Other closely relatedwork used epipolar geometry for bronchoscope localizationin CT [14]. The most closely related work to our own waspresented by Luo et al. [15] in which a system was developedfor bronchoscopic tracking for navigation. Again, the goals of

the work are different. In lung biopsy, the goal is for long-range navigation to successfully follow the bronchial tree to atarget location, whereas the goal of our work is short-range re-registration for high-accuracy visualization of critical anatomy.Furthermore, our work has a number of clear differences, (i)the use of SVD-Match for correspondence initialization, (ii)the use of ASKC for motion estimation, (iii) the use of feature-based registration to CT [16] and (iv) the continued use ofASKC for tracking.

II. METHOD

There are five major components in our system. Fig. 2shows an overview of the system with the five componentshighlighted in blue. First, the system extracts Scale InvariantFeature Transform (SIFT) features [17] from the video data.Next, the motion between the images is estimated, after whichfeature points are reconstructed. At the beginning of the videosequence, reconstructed points from the first pair of imagesare registered to an isosurface segmented from CT data. Thisinitial registration is used to initialize a registration trackingalgorithm that makes use of feature matches in subsequentframes. Each of these components is detailed in the followingsubsections.

A. Feature detection and matching

In the first step, video data are processed to extract imagefeatures using the Matlab (The Mathworks Inc., Natick, MA,USA) implementation [18] of SIFT features [17]. Previous work[19] has shown SIFT performs well on endoscopic video. Wetested both the SIFT feature matching technique suggested byLowe and the SVD SIFT matching technique [12] originallypurposed for two-dimensional point features in [20]. SVDSIFT computes the normalized cross-correlation weighted bythe distance between all the features. The SVD is applied tothe weighted correlation matrix to normalize the feature matchscores by setting the singular values to 1. This up-weights weakmatches and down-weights strong matches in turn, increasing

3

Video/CT Registration

Reconstruction Tracking Display EndoscopicVideoFeature

ProcessingMotion

Estimation

SegmentationCT

Fig. 2: System Overview: First the endoscope images are summarized by extracting and matching features. The motion isestimated and geometry reconstructed, after which the CT is registered and the video tracked. Each of the boxes highlighted inlight blue are described in sections II-A through II-E.

Needles

Needles

(a) (b) (c) (d)

Fig. 3: Endoscope image of the sphenoid sinus. (a) the undistorted endoscope image from a cadaver, (b) the detected featuresin an endoscope image, (c) the initial correspondence from SVD SIFT with inliers (blue) and outliers (red) and (d) the result ofapplying the robust ASKC method to estimate the motion.

the number of matches at the possible cost of lowering theaccuracy of the matches. A pair of features is considereda match if the pair is greater than 50% correlated and thefeatures within 100 pixels of each other. As we previouslydemonstrated in [21], SVD SIFT provides a larger numberof matches (generally in the hundreds), which increases thenumber of points in the reconstruction. Thus, we used SVDSIFT for initial matching. Fig. 3 shows the detected features(3b) and initial correspondence from SVD SIFT, which includesinliers (blue) and outliers (red) (3c).

B. Motion Estimation with ASKC

After the image features are detected and matched, theendoscope motion is estimated using the robust techniqueof Wang et al. [22]. To solve the motion estimation problemwe consider the following setup. When a camera observes a3D point X on a surface from two distinct positions, the pointX will project to two image locations x1 = (u1, v1, 1)T andx2 = (u2, v2, 1)

T . Let R and t represent the motion betweenimages. It is well known that the following condition holds[23]:

x2TK−T2 sk(t)RK

−11 x1 = 0, (1)

where K1 and K2 are the intrinsic camera matrices corre-sponding to the two images, sk(t) is the skew matrix of thetranslation vector t and R is the rotation matrix. The essential

matrix E = sk(t)R encodes the motion information of thecamera. Given the intrinsic camera matrices, the essential matrixE can be estimated using a nonlinear five-point algorithm [24].The camera motion (R and t) can be recovered from E bythe SVD approach [25], although the translation can only beestimated up to a scale factor. In what follows we use γ̃ torepresent the estimated scaled translation vector and γ = λγ̃.The scale λ can be recovered by registering the reconstructed3D model to a pre-operative CT scan [10].

To robustly estimate the camera motion from n matches, weemploy an Adaptive Scale Kernel Consensus (ASKC) estimator[13]: [

t̂, R̂]

= argmaxt̂,R̂

n∑i=1

1

hiker

(rihi

), (2)

where ker(·) is a kernel function. The kernel function we usein this work is the Gaussian kernel, which was previouslyshown in [13] and [22] to work well for this problem. Thevariables hi and ri are respectively the bandwidth estimateand the residual of the ith matches. Since ASKC is a variable-bandwidth technique, hi is automatically estimated from thedata. In this case ri is the following:

ri(t, R,K1,K2;x(1,i),x(2,i)) = xT(2,i)K

−T2 sk(t)RK

−11 x(1,i).

(3)Fig. 3d shows an example where ASKC has correctly estimatedboth the epipolar geometry and the scale of inliers, as well

4

as selected the correct matches even though the percentageoutliers was at times larger than 70%.

C. ReconstructionOnce motion estimation is complete, the motion information

(R,t) is used to reconstruct the 3D structure up to scale. Thefirst pair of images is reconstructed with triangulation andsubsequent pairs of images are reconstructed from the trackedimage features. The image features are tracked using SVDSIFT feature matches by matching feature of pairs of imagesand building feature trajectories.

We used a calibrated camera and removed the opticaldistortion with [26]. Let Xi =

[xi yi zi 1

]Tbe a 3D

point in the world coordinate system. The 3D point Xi isprojected to an image point xfi =

[ui vi 1

]Tat viewing

position, or frame, f by a 3x4 projection matrix Pf .

xfi = PfXi (4)

Let the first camera be at the origin of the world coordinatesystem, and we have:

P1 = K [I|0] and Pf = K[1Rf |1tf

], (5)

where 1Rf and 1tf are respectively the rotation and thetranslation of the camera at the f th frame relative to those ofthe camera at the first frame. Note that the camera matrix Kof the endoscope remains fixed throughout the sequence. Atthe beginning, the structure is initialized using two frames bytriangulation [25]. These two frames are selected by using themethod described in [27] or by manual selection. For a newframe f , we relate it to its previous frame f − 1. Assumingwe have known Pf = K

[1Rf−1|1tf−1

]at the frame f − 1,

Pf can be written as:

Pf = K[1Rf−1

f−1Rf |1Rf−1f−1tf + λf 1t̃f−1]. (6)

Let:Cf =

1Rf−1f−1tf ,K = [k1,k2,k3]

T, (7)

and

Pf =

pf11 pf12 pf13 pf14pf21 pf11 pf23 pf24pf31 p

f32 p

f33 p

f34

. (8)From equations (4), (6), (7) and (8), we can derive:

ufi =pf11xi + p

f12yi + p

f13zi + k1Cf + λfk1

1t̃f−1

pf31xi + pf32yi + p

f33zi + k3Cf + λfk3

1t̃f−1

vfi =pf21xi + p

f22yi + p

f23zi + k2Cf + λfk2

1t̃f−1

pf31xi + pf32yi + p

f33zi + k3Cf + λfk3

1t̃f−1

. (9)

If we define the following:

Afi =

[ufi k3

1t̃f−1 − k11t̃f−1ufi k3

1t̃f−1 − k21t̃f−1

], (10)

and

Bfi =

[pf11xi + p

f12yi + p

f13zi + k1Cf − u

fi pf31xi

pf21xi + pf22yi + p

f23zi + k2Cf − v

fi pf31xi

+ufi pf32yi + u

fi pf33zi + u

fi k3Cf

+vfi pf32yi + v

fi pf33zi + v

fi k3Cf

] ,(11)

we can calculate the scale value λf by:

λf (i) =(ATf,iAf,i

)−1ATf,iBf,i. (12)

However, as both the feature’s location and the 3D points maybe in error, we estimate λf in a robust way:

λ̂f = argmaxλf (i)

1

n

n∑j=1

1

hjker

(rjhj

), (13)

where rj =∑∣∣∣xfi − Pf (λf (i))X̂i∣∣∣ and hj is estimated from

the data with the robust k scale estimator [28]. After P̂f isestimated, the 3D points {Xi} having correspondences to thetracked SIFT features are refined:

X̂i = argminX̂i

wi−1∑j=0

∑∣∣∣xf−ji − P̂f−jX̂i∣∣∣ . (14)Newly appearing 3D points are initialized and added to thestructure. Algorithm 1 gives an outline of the reconstructionalgorithm.

Algorithm 1 ({Xi}i=1,...,m, {Pj}j=1,...,n) = Reconstruction({Imagesj}j=1,...,n)

1: for all Imagesi do2: Extract SIFT features using SIFT detector [17]3: //Initialize the 3D structure.4: Choose two initial frames and detect potential matches

by the SVD SIFT algorithm [12].5: Select the correct matches by ASKC (2) and calculate

the motion parameters of the endoscopic camera.6: Initialize the structure {Xi} by triangulation [25].7: //Maintain the 3D structure.8: Obtain matches between the frames f and f − 1.9: Track the SIFT features using the feature tracking

algorithm proposed in [22]10: Compute the projection matrix Pf by (5)11: Compute the 3D points corresponding to the new SIFT

features and add them to the 3D structure.12: Refine the existing 3D points that correspond to the

tracked SIFT features.13: end for14: Output the reconstructed 3D structure {Xi}i=1,...,m and

the projection matrices {Pj}j=1,...,n.

D. Registration

The reconstructed 3D point cloud is registered to a surfacesegmented from a CT image of the same patient. The surfaceis segmented by applying a threshold at the air/tissue boundary(approximately -500 Hounsfield Units) and post-processedusing marching cubes [29] to create a polygon isosurface.We applied a registration algorithm described in [16] that isderived from Trimmed ICP (TrICP) [30] and extends TrICPwith scale [31]. Since the true scale of the 3D world is lostin the epipolar constraint (1), the registration algorithm alsoneeds to estimate the scale of the 3D model.

5

Our registration algorithm requires three inputs: a 3D pointcloud of relative scale with the origin at the camera center, anisosurface from the CT, and an initial estimate of the scaleand location. The 3D point cloud is the expected output of the3D reconstruction process. We assume that the 3D point cloudis of uniform scale that need not be the same as the CT, andthat the origin of the point cloud is the camera center suchthat the rigid-body transformation aligning the 3D point cloudis the camera location in CT coordinates. Outliers typicallycontaminate the 3D reconstruction as a result of mismatchedfeatures and the ambiguity of sign in the epipolar constraint.

The second input, the isosurface from the CT, is essentialbecause the algorithm registers 3D points to a surface. Whileusing only a surface does remove a large amount of data, thereis sufficient data in the surface alone for registration. Similar toICP [32], TrICP is prone to local minima and an approximatesolution is needed to start the algorithm, thus the third input,an initial estimate of the location and scale is required.

TrICP was chosen for its robustness to outliers and simpledesign. Our algorithm modifies the traditional TrICP algorithmby adding scale. This modification is similar to the work of Duet al. [31] However, we assume a single uniform scale. Thefollowing is the derivation of the modified TrICP algorithm.

Let X ∈

6

Algorithm 4 ({Rj}j=1,...,n,{tj}j=1,...,m) = Camera PoseTracker (Rinit, tinit,sinit,{Xi}i=1,...,m, {Imagesj}j=1,...,n,mesh)

1: R← Rinit, t← tinit, Xcurrenti ← sinitXi2: for j ← 2 to n do3: if j = 2 then4: ̂Image1 ←undistort( Images1)5: sift1 ← detect SIFT feature( ̂Image1)6: else7: ̂Image1 ← ̂Image28: sift1 ← sift29: end if

10: ̂Image2 ←undistort( Imagesj)11: sift2 ← detect SIFT feature( ̂Imagej)12: matches = SVDSIFT match(sift1, sift2)[12]13: (E, inliers) = robustMotionEstimator(matches) (2)14: //Track the old sift feature and add new matches.15: {Xcurrenti } = tracker(matches, inliers, {Xcurrenti } )

[22]16: (R̂, t̂) = robustPoseEstimator(Xcurrenti ,matches,R,t)

(2)17: {XreprojectedInliersi } = reprojectPoints( {Xcurrenti }, R̂,

t̂ )18: (R, t) = robustPoseEstimator( {XreprojectedInliersi },

matches, R̂, t̂ ) (2)19: //Remove outliers from refined points20: {Xrefinedi } = refinePoints( {X

reprojectedInliersi },

mesh ) (19)21: for all previous Ri, ti do22: (Ri, ti) = robustPoseEstimator( X

refinedi ,

matchesi,Ri, ti) (2)23: end for24: end for

The system first undistorts the images, then SIFT features aredetected and matched. After finding the set of matched imagefeatures, these image feature points are used to estimate themotion of the frame pair. The inliers of the motion estimationare then tracked using the SIFT feature tracking method from[22]. Once the new camera pose is estimated, we refine boththe 3D points and pose estimates. The 3D points are refinedby applying all of the previously seen image features andprojection matrices and then solving the following null spaceproblem. First, given a simple projection model from (4), letRi =

[rT1,i r

T2,i r

T3,i

]Tand ti =

[t1,i t2,i t3,i

]T. The

3D point which was created by all of the imaged points is thenull space solution of

0 =

rT1,1 t1,1 − u1 rT3,1 t3,1rT2,1 t2,1 − v1 rT3,1 t3,1

...

x1y1z11

. (19)III. IMPLEMENTATION AND EXPERIMENTS

The algorithm was prototyped in Matlab. After the initialevaluation previously reported in [22], [16], and [21], we

translated some portions of the code into C++ and merged thecode into the TREK image-guided surgery software platform[35]. The Matlab code was directly merged into Python [36]for use with TREK via mlabwrap [37]. The TREK platformenabled easy access to the visualization components of 3DSlicer [38] and the processing components of the cisst library[39].

A. Experimental Setup

For our experiments we used both a skull phantom andcadaveric human head. Both were set up similarly.

1) Phantom: A skull phantom was used to perform theexperiments in a dry lab. A Sawbones (Pacific ResearchLaboratories, Vashon, Washington, USA) skull phantom wasprepared with #4 and #6 aluminum sheet metal screws.Aluminum, having a low atomic number, produces minimalartifacts in CT scans. Twelve of the larger #6 screws wereplaced in the front and sides of the cranium as shown in Fig.4. The smaller #4 screws were placed approximately where thesphenoid sinus would be. Though the phantom did not havea sphenoid sinus, it enabled us to mock up the geometry ofthe entire skull and nasal passage. The #6 screws were usedto register the skull to the navigation system during the videorecording. The skull phantom was mounted to a stand andclamped to the work bench. An Optotrak rigid-body was alsoattached to the work bench. The skull phantom was finallyregistered to the navigation system using a ball pointer tool.

2) Cadaver: The protocol for the phantom study was thentransfered to the wet lab in our cadaver study. Similarly, ten #6screws were implanted in the front and sides of the craniumof the cadaver head. After a craniotomy was performed, 27gauge syringe needles were placed in the sphenoid sinus wall,the pituitary gland, carotid arteries and optic nerves, shown inFig. 5. After implanting the needles into the bone or tissue,Great Stuff (Dow Chemical Company, Midland, Michigan,USA) was used to secure and support the needles. Each of theneedles were used as a target structure or critical structure thatrequires high-precision navigation to identify. Again the #6screws were used to register the navigation system to the videoduring recording. The cadaver head was secured in a Mayfieldclamp. We performed the cadaver experiment in two differentspecimens. In our first study, we examined navigation error(defined below in section III-D) versus TRE. The second studywas used to compare pointer-based, tracker-based and video-based navigation. Table I shows a summary of the differentmethods.

Method Name DescriptionPointer-based Targets are identified and localized with a

pointer tool.Tracker-based Targets are identified and visualized in the

video registered to the CT via a trackingsystem

Video-based (Our Method) Targets are identified and visualized inthe video registered to the CT via videoreconstruction and registration

TABLE I: A summary of three methods that will be comparedin our experiments.

7

#6 Screws

#4 Screws

(a)

#4 Screws

(b) (c)

Fig. 4: Distribution of screws (a,b) and CT slices with an endoscope image (c) of the skull phantom.

#6 Screws

May�eld Clamp

Needles

Great Stu�

(a)

#6 Screws

Great Stu�

Needles

May�eld Clamp

(b) (c)

Fig. 5: Cadaver: Shown in (a) and (b) are the distribution of screws and needles with Great Stuff in the cadaver. (c) shows CTslices and an endoscope image.

Optotrak Rigid-bodies

EndoscopeCamera

(a) Endoscope and Optotrak rigid-body

Optotrak

EndoscopeTower

StealthStation

Navigation

Phantom

Pointers

(b) Experimental System Setup

Fig. 6: Shown in (a) is the configuration of Optotrak rigid-bodies on the camera and endoscope. Presented in (b) is an overviewof the experimental system setup, showing each of the tracking systems (Optotrak and StealthStation) used and their relationshipto the phantom.

8

B. Data Collection

We collected endoscopic phantom sinus and ex-vivo humansinus video data. The video was captured at 1024× 768, 30hz using a Flea2 (Point Grey Research, Richmond, BritishColumbia, Canada) firewire camera attached to a zero-degreerigid monocular Storz (Karl Storz, Tuttlingen, Germany) endo-scope. Optotrak rigid bodies were attached to the endoscopeand specimen, shown in Fig. 6. The Optotrak motion datawere used to initialize the endoscopic motion by transformingthe motion of the rigid body attached to the endoscope tothe camera center by way of the hand-eye calibration. TheOptotrak and Flea2 were synchronized via the triggering pinon the Optotrak. In this way the clock signal the camera emitswas the clock for the Optotrak. To verify that the video andtracker motion were synchronized, video of a pure translationwhile viewing a fixed point was recorded. Before the datacollection, images of a checkerboard calibration grid were alsorecorded using the endoscope. We performed an offline cameraand hand-eye calibration of the endoscope using the GermanAerospace Center (DLR) Camera Calibration Toolbox [40].The CT data used had 0.46 mm × 0.46 mm × 0.5 mm voxels.

C. Sources of Error

Our data collection had four potential sources of error: 1) thecamera calibration; 2) the Optotrak to camera calibration; 3)the Optotrak to CT registration; and 4) unaccounted for motion.The camera was calibrated within an error of [0.38, 0.37] pixels.The aforementioned Optotrak/camera configuration had anestimated position uncertainty in millimeters of the cameracenter as measured by the Optotrak of [1.1, 0.4, 0.1]. TheOptotrak to CT registration was generally within .5 mm RMSpoint distance error. Each of these contributes to an overalllocation uncertainty of approximately 1.5 mm in the absoluteposition of the endoscope optical center, and approximately1.1 mm relative position accuracy.

D. Evaluation Methodology

We compared our method to both pointer-based navigationand tracker-based endoscopy using three error metrics. Themetrics we considered were two target registration errors anda metric we named navigation error (NGE), defined below. Wedefine TRE1 as described by Fitzpatrick and West [41] forthe traditional pointer-based approach.

TRE1 =∥∥pCT − (CTTNavigation)ppointer∥∥ (20)

Where pCT is the target segmented from the CT,CTTNavigation is the transformation from the tracking systemto the CT as computed with the fiducial points and ppointeris the current pointer location.

For both tracker-based endoscopy and video-based registra-tion, we defined TRE2 as described in Mirota et al. [42]

r = RK−1pimage − t, (21)

where R, K and t are as defined in (5) and pimage is thepoint segmented in the video image. Equation (21) defines aray from the camera center (t) in the direction of the target in

Metric Name DescriptionTRE1 Metric for evaluating pointer-based methodsTRE2 Metric for evaluating tracker-based and

video-based methodsNGE Same as TRE2, however, the target is not

visible in the endoscope image.

TABLE II: A summary of the metrics used in our experimentalevaluation.

the image. If the camera was exactly registered, the ray wouldpass through the center of the target in the CT. We adopt thismetric to be able to directly compare tracker-based endoscopywith our video-based registration, because only camera andimage data are available in tracker-based endoscopy. The ray(r) is then projected to the closest point to the target (pCT )which is used to compute the TRE as follows:

TRE2 =

∥∥∥∥pCT − (t + r(r · (pCT − t)r · r))∥∥∥∥ . (22)

Another way to think about (22) is that it creates a virtualpointer along the ray from the endoscope with end point equalto the closest point to the target along the ray. The majordifference in the two TREs are that in the first case 3D pointdata are compared directly, whereas in the second case a cameralocation and ray are used to measure the error in the cameraimage.

We defined NGE as the relative distance between a targetsegmented in the CT data and the location the navigationsystem reported as the location of the target. Mathematically,NGE is the same as (22). NGE is distinct from TRE in that thetargets are not visible in the endoscope and only the surgeon’sperception and the navigation system guide the location of thetarget. In both TREs the targets are clearly visible in both theendoscope and the navigation system. Table II summarizes theerror metrics we applied in our evaluation.

For context, we measure traditional pointer-based TRE andtracker-based endoscopy TRE to show the differences betweenour method and that which is currently available in the operatingroom as well as a method similar to that of the state of the artin tracker-based endoscopy [8]. For the tracker-based resultsthe Optotrak was used to record camera motion. Additionally,we initialize the camera location in the video-based methodwith the pose of the camera measured by the Optotrak. TheStealthStation could be used instead of the Optotrak to providean initialization. To avoid single sample basis we took 10measurements and recorded the error for each target that wasvisible. We then averaged over the samples of a target toproduce a measurement at a single target.

1) Statistical Evaluation of TRE: For further statistical evalu-ation of TRE, we assume that the three orthogonal componentsof both TRE1 and TRE2 are independent, normally disturbedvariables with zero mean. The expectation of TRE2 will beof the form:

〈TRE2〉 = σ21 + σ22 + σ23 , (23)

where σ21 , σ22 , and σ

23 represent the variances of the three

component random variables. Our estimate of 〈TRE2〉 is

9

therefore a measure of total variance, so we test the nullhypothesis that the expected mean TRE2 values are equal toelucidate the relative performance of our navigation systems.Since TRE2 represents the sum of three chi-squared variables,its distribution is not normal. We use the zero-skewness logtransform to normality to account for this. We used a mixedlinear model for data analysis. For the cadaver head study,we account for three major sources of variation: i) a headterm which accounts for experimental variability caused bydifferences between heads; ii) a pin term which accountsfor experimental variability due to differences between pins;this accounts for differences between target registration errorbetween pins due to the relationship between target registrationerror and fiducial registration error and variability between thesurgeons’ ability to touch pins in the nose due to anatomicconstraints of the cadaver specimens; and iii) residual error.Residual error will be a function of the scatter in data causedby the collection method, i.e., test subjects touching pins with apointer, in addition to the traditional sources of error attributedto regression analysis. The pin term is nested within the headsterm. To this random effects model we added three fixedcovariates to elucidate the effects of navigation methods ontarget registration error. We assumed the Optotrack representedthe gold standard and use it as our reference.

We used a similar regression model to analyze the phantomdata. Since we only performed one phantom study, we droppedthe head term from the phantom data analysis. To compare thenavigation systems, we calculate the marginal means treatingthe factor variables as balanced and then perform post hoc Waldtests corrected for multiple comparisons using Bonferroni’smethod. The post hoc tests perform hypothesis tests on thedifferences in transformed squared errors between methods. Forcadaver head one we also compared TRE2 to NGE2 for thetracker based and video-based navigation systems. We used atwo-factor analysis of variance (ANOVA) for this comparison.The squared errors served as the dependent variables. Twofactors (navigation method and error type) with two levelseach (tracker-based and video-based; TRE2 and NGE2) wereused as dependent variables. The two factors were allowedto interact. We set the global type I error at 0.05. Since wecompared TRE2 to each other and TRE2 to NGE2 for eachfor the cadaver study we use an alpha of 0.025 for each test.

2) Evaluation of Tracking Accuracy: Since we previouslyestablished that the absolute position of the Optotrak ispotentially not as accurate as video tracking, we verify ourmethod has similar relative trajectory accuracy by evaluatingrelative motion difference.

Fdifference =[ROi tOi0T 1

]−1 [Ri ti0T 1

] [R1 t10T 1

]−1 [RO1 tO10T 1

](24)

The variable i is from 1 to the number of images of thesequence. Using the cadaver data, we evaluated 10 arbitrarysegments of video consisting of 67 images each.

3) Registration Initialization Sensitivity: Since we used thetracker to initialize the registration process, we evaluated theeffects of errors in the tracker data. We introduced errors intothe tracker by adding an in-plane or out-of-plane perturbation

vector with varying magnitude from 0 to 5 mm. The in-planemotion was defined by the estimated pose of the imaging planeand out-of-plane motion is any motion perpendicular to theimaging plane.

4) Evaluation of Shape Accuracy: To evaluate the accuracyof the shapes from the reconstruction process we examinedthe mean distances to the CT surface.

IV. RESULTS

The following is a discussion of the TRE measured in eachof the experiments. In all of the following box plots the rankstatistics from top to bottom are maximum, third quartile,median, first quartile and minimum.

A. Synthetic Data

1) Tracking: We focused our evaluation on tracking. Weevaluated the robustness of the method to noise and outliersby creating projected 2D points of a known 3D model, thenadding uniform noise or uniform outliers to the projected 2Dpoints. We assumed the registration to the model was perfect toisolate the analysis to only tracking. Fig. 7 shows the error oftracking in each frame of the 10 frames used in the simulation.Translation error is the norm of the translation componentof the motion difference. Rotation error is norm of the Eulerangles of the rotation component of the motion difference. Thetests on synthetic data showed that the tracking method wasrobust to large outliers and was very robust to noise. It is clearthat noise has little effect on performance. Since the pointswere sampled with ASKC, it quickly rejects both outliers andpoints with large noise. Additionally, it can be seen that thetracker drifts in the presence of high noise (greater than 60pixels) or high percentage of outliers (greater than 40%).

B. Phantom Study

Fig. 8 summarizes the overall distribution of all TREmeasurements within the rigid Sawbone phantom. The Optotrakpointer and Medtronic StealthStation pointer provide contextfor the tracker-based and video-based methods. The Optotrakpointer, the gold standard in tool tracking, showed the highestaccuracy and precision, as expected. The mean TRE of theOptotrak information was 0.51 mm (with 0.44 mm first quartileand 0.38 mm range), while the StealthStation mean TREwas 2.56 mm (with 1.73 mm first quartile and 2.16 mmrange). The mean TRE for the tracker-based method was 1.30mm (with 1.28 mm first quartile and 1.49 mm range). Bycomparison, the mean TRE for the video-based method was0.98 mm (with 0.83 mm first quartile and 0.98 mm range).The improvement in TRE over the tracker-based method wasstatistically significant (p < 0.01), with the majority of alltargets having sub-millimeter precision.

C. Cadaver Studies

In the first cadaver study, the navigation error and TREwere measured. Fig. 9 shows the distribution of both metricsin the study. The navigation error was greater than the TRE,which was expected because the needle tips are hidden from

10

1 510

050

1000

0.01

0.02

Frame Numb

er

Translation Error with Noise

Noise (pixels)

Erro

r (m

m)

(a)

1 510

050

1000

0.1

0.2

Noise (pixels)

Erro

r (de

g)

Rotation Error with Noise

Frame Numb

er

(b)

1 510

0204060

0

1

2

Erro

r (m

m)

% Outliers

Translation Error with Outliers

Frame Numb

er

(c)

15

10020

40600

10

20

Erro

r (de

g)

% Outliers

Rotation Error with Outliers

Frame Numb

er

(d)

Fig. 7: Results from our tracking method with synthetic data in which the effects of outliers and noise on tracking performancewere characterized. (a) Effect of noise on the translation component of the motion. (b) Effect of noise on the rotation component.The effects of outliers and nominal noise on translation and rotation are shown in (c) and (d).

OptotrakPointer

StealthStationPointer

Tracker-based

Video-based

0.5

1

1.5

2

2.5

3

3.5

Phantom Study TRE

TRE

(mm

)

Fig. 8: Overall distributions of TRE for the phantom study.Our method is labeled video-based. The rank statistics fromtop to bottom are the, third quartile, median, first quartile andthe minimum.

view and not protruding into the sinus. In the navigation errorcase, the tracker-based registration method mean error was2.36 mm (with 1.71 mm first quartile and 3.09 mm range),while the video-based registration had a mean error of 1.54mm (with 1.13 mm first quartile and 1.87 mm range). Whenlooking at TRE, the video-based method (with a mean of 1.25mm, 0.96 mm first quartile and 1.51 mm range) outperformedthe tracker-based method (with a mean of 1.43 mm, 1.28 mmfirst quartile and 0.77 mm range). The results demonstratethat video-based registration is moderately more accurate thantracker-based methods.

Shown in Fig. 10 is the overall distribution of TRE measuredin the cadaver. Again, for context, the Optotrak pointer andMedtronic StealthStation pointer are shown for the targets. TheOptotrak had a mean TRE of 1.25 mm (with 1.14 mm firstquartile and 0.93 mm range). The StealthStation was closerto the Optotrak in the experiment with a mean TRE of 1.49mm (with 1.28 mm first quartile and 1.43 mm range). For thetracker-based registration method, the TRE was 1.01 mm (with0.93 mm first quartile and 0.72 mm range). By comparison,the mean TRE for the video-based method was 0.70 mm (with

Erro

r (m

m)

Comparison of NGE vs. TRE in Cadaver Study 1

NGE TRE

Tracker-based

Video-based

Tracker-based

Video-based

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Fig. 9: Overall distributions for cadaver study #1 comparingnavigation error (NGE) and TRE. Our method is labeled video-based. The rank statistics from top to bottom are maximum,third quartile, median, first quartile and minimum.

0.56 mm first quartile and 0.50 mm range). The results againdemonstrate a significant improvement (p < 0.001) for thevideo-based registration approach with all but one target withsub-millimeter precision.

In Fig. 11 we compare the overall distribution of TREmeasured in both cadaver studies. Both studies exhibit a generaltrend of improvement, although there is a clear difference inthe range of the video-based method between the two cadaverstudies. In the first study the maximum of the video-basedmethod (max 2.16 mm) was greater than the tracker-basedmethod (max 1.88 mm). The difference was caused by twosamples of the target with maximum error in which the scaleof the 3D point cloud was not estimated correctly because ofmotion blur in the original images. We prevented motion blurin the second cadaver study by adjusting the gain, exposureand shutter settings on the camera.

Finally, Fig. 12 presents an example registration resultshowing how the video-based method brings the targets intocloser alignment than the tracker-based method.

11

(a) Tracker-based TRE

TRE(mm)1.50

0.00

0.75

(b) Tracker-based TRE Colormap (c) Video-based TRE

TRE(mm)1.50

0.00

0.75

(d) Video-based TRE Colormap

Fig. 12: Example registration result shows the TRE in the video. In (a-d) needle targets (placement of which is shown in Fig.3a) in the video are labeled with solid green circles and the respective targets in the CT are labeled with solid blue circles

OptotrakPointer

StealthStationPointer

Tracker-based

Video-based

0.6

0.8

1

1.2

1.4

1.6

1.8

2

2.2

TRE

(mm

)

Cadaver Study TRE

Fig. 10: Overall distributions for cadaver study #2. Our methodis labeled video-based. The rank statistics from top to bottomare the maximum, third quartile, median, first quartile and theminimum.

Tracker-based

Video-based

Tracker-based

Video-based

Study #2Study #1

0.5

1

1.5

2

TRE

(mm

)

Comparison of Cadaver Studies

Fig. 11: Comparison of the overall distributions for the cadaverstudies. Our method is the video-based column. The rankstatistics from top to bottom are the maximum, third quartile,median, first quartile and the minimum.

D. Statistical Evaluation

In the following tables CI stands for Confidence Interval. Thek parameter for the log skew normality transformation for theTRE2 cadaver head data was −0.109. The regression resultsare displayed in Table III and Table IV. The marginal means asa function of navigation method are shown in Table V. We notethat the Optotrak had the largest marginal mean and thereforethe highest total variance while the video-based navigation hadthe lowest overall marginal mean and therefore the smallest totalvariance. Post hoc testing (Table VI) shows that the difference intotal variance between the Optotrak and StealthStation was notstatistically significant; all other comparisons were statisticallysignificant, suggesting that tracker-based navigation performsbetter than traditional navigation systems and that video-basednavigation performs best.

TABLE III: Head Study Fixed Effects Results.

Covariate RegressionCoefficient

StandardError

p-value 97.5% CI

Stealth-Station -0.23 0.51 0.66 (-1.369, 0.916)Tracker-Based -1.03 0.4 0.01 (-1.945, -1.115)Video-Based -1.56 0.4 < 0.001 (-2.475, -0.645)

Constant 1.32 0.57 0.02 (0.487, 2.596)

TABLE IV: Head Study Random Effects Results.

Random EffectsParameters

Estimate StandardError

97.5% CI

SD: Head Term 0.68 0.49 (0.132, 3.471)SD: Pin Term 0.31 0.07 (0.179, 0.526)SD: Residual 0.76 0.03 (0.703, 0.828)

TABLE V: Head Study Marginal Means.

Navigation MarginalMeans

StandardError

p-value 97.5 % CI

Optotrak 1.13 0.49 0.02 (0.033, 2.231)Stealth-Station 1.11 0.49 0.02 (0.004, 2.216)Tracker-Based 0.29 0.49 0.55 (-0.808, 1.389)Video-Based -0.25 0.49 0.6 (-1.352, 0.844)

12

TABLE VI: Head Study Hypothesis Testing a

Optotrak StealthStation Tracker-BasedStealthStation χ2(1) = 0.04; p = 1.000Tracker-Based χ2(1) = 59.59; p < 0.001 χ2(1) = 43.83; p < 0.001Video-Based χ2(1) = 161.72; p < 0.001 χ2(1) = 121.48; p < 0.001 χ2(1) = 26.80; p < 0.001

a Post Hoc testing, corrected for multiple comparisons with the Bonferroni method.

The k parameter for the log skew normality transformationfor the NGE2 was −0.345. Both the navigation methodterm (F (1, 154) = 9.16; p < 0.001) and the error term(F (1, 154) = 9.16); p = 0.003) were statistically significant.The interaction term was not significant. Post hoc testing showsthat all of the ANOVA contrast comes from the fact that thetracker-based NGE2 is significantly larger than all other errors.

The k parameter for the log skew transformation to normalityfor the TRE2 phantom data was −0.068. The regression resultsare show in Tables VII and VIII. The marginal means areshown in Table IX. In the phantom study the Optotrak had thelowest total variance and the Stealth station had the greatesttotal variance. The video-based navigation and tracker-basednavigation were in between with the video-based navigation’svariance being smaller than the tracker based navigation. Posthoc testing shows that all differences in total variance werestatistically significant (see Table X).

TABLE VII: Phantom Study Fixed Effects Results.

Covariate RegressionCoefficient

StandardError

p-value 95% CI

StealthStation 2.47 0.18 < 0.001 (2.108, 2.823)Tracker-Based 1.73 0.17 < 0.001 (1.389, 2.073)Video-Based 0.91 0.17 < 0.001 (0.566, 1.249)

Constant -1.15 0.13 < 0.001 (-1.403, -0.898)

TABLE VIII: Phantom Study Random Effects Results.

Random EffectsParameters

Estimate StandardError

95% CI

SD: Pin Term < 0.001 < 0.001 (0, 0.00004)SD: Residual 1 0.04 (0.905, 1.101)

TABLE IX: Phantom Study Marginal Means.

Navigation MarginalMeans

StandardError

p-value 95% CI

Optotrak -1.15 0.13 < 0.001 (-1.403, -0.898)StealthStation 1.31 0.13 < 0.001 (1.062, 1.567)Tracker-Based 0.58 0.12 < 0.001 (0.350, 0.811)Video-Based -0.24 0.12 0.04 (-0.474, -0.013)

E. Tracking with Cadaver Data

Table XI shows the relative motion difference betweenthe Optotrak and the tracking methods. It is worth noting

that after the initialization, no Optotrak information wasused while tracking. A majority of the segments show sub-millimeter differences. Though three segments were greaterthan a millimeter, tracking was never lost over the course ofall of the video. In Fig. 13, we display an example Optotraktrajectory and the corresponding video-based tracking trajectory.The two agree nearly completely with the exception of adifferent starting location, which was the result of the video-based registration.

TABLE XI: Mean pose error and standard deviation of thetrajectories

Trans. Difference (mm) Rot. Difference (deg)Segment 1 1.54 (1.04) 0.05 (0.03)Segment 2 1.68 (0.80) 0.04 (0.02)Segment 3 0.78 (0.30) 0.06 (0.03)Segment 4 0.93 (0.81) 0.03 (0.03)Segment 5 0.90 (0.54) 0.03 (0.01)Segment 6 0.87 (0.59) 0.03 (0.02)Segment 7 0.66 (0.28) 0.03 (0.01)Segment 8 2.70 (1.23) 0.19 (0.10)Segment 9 0.73 (0.74) 0.03 (0.06)Segment 10 0.70 (0.30) 0.04 (0.01)

Mean Difference 1.15 0.05

Video−basedTracker−based

−5−4

−3−2

−10

12

15.516

16.517

17.5

−507

−506

−505

−504

−503

−502

3325

33

17

25

1

41

49

9

57 117

4157

49

9

65

65

3325

33

17

25

1

41

49

9

57 117

4157

49

9

65

65

1717

3333

Tracker−basedVideo−based

Fig. 13: An example tracked trajectory with Optotrak in blueand our video-based method in red.

F. Initialization Sensitivity

Fig. 14 is a comparison of the Optotrak location TRE within-plane and out-of-plane pertubation and the resulting video-

13

TABLE X: Phantom Study Hypothesis Testing a

Optotrak StealthStation Tracker-BasedStealthStation χ2(1) = 182.99; p < 0.001Tracker-Based χ2(1) = 98.45; p < 0.001 χ2(1) = 17.70; p < 0.002Video-Based χ2(1) = 27.05; p < 0.001 χ2(1) = 79.70; p < 0.001 χ2(1) = 24.52; p < 0.001

a Post Hoc testing, corrected for multiple comparisons with the Bonferroni method.

based registration TRE. The red and blue lines are the Tracker-based and the Video-based median TRE, respectively. The errorbars are the first and third quartiles of the TRE distribution andthe line, the median. As we mention in section III-D1, TRE isnot normally distributed, so again we used the zero-skewnesslog transform to normality and a two-sample Student’s t-test totest the difference of the distribution. We applied Bonferroni’scorrection once more to account for multiple tests. The video-based registration has a significant improvement for a majorityof the perturbations in-plane. A single star ( * ) indicatesp < 0.0024, α = 0.05/21, and no stars indicate no significantdifference (p > 0.0024, α = 0.05/21). In Fig. 14 trendregions between the tracker-based and video-based TRE areevident, indicating that the scene geometry was not sufficientto minimize the registration further. In both the in-plane andout-of-plane motion cases there are minima at -1 mm, revealingthe inherent 1 mm error of the Optotrak measurements. Alsoin Fig. 14b beyond 2 mm the scale was not solved correctly.We conclude that for best performance a tracking system withat least 2 mm accuracy would be needed to initialize theregistration to ensure sub-millimeter registration and tracking.

G. Shape Accuracy

We observed that the shape reconstructed agreed well withthe CT surface and consistently fit within 1 mm. Table XIIdetails the result of the reconstruction for each of the segmentsused for tracking.

TABLE XII: Mean distance to the surface of the CT

Mean Distance (mm)Segment 1 0.30Segment 2 0.36Segment 3 0.27Segment 4 0.16Segment 5 0.18Segment 6 0.20Segment 7 0.20Segment 8 0.22Segment 9 0.25Segment 10 0.22

Mean 0.24

H. Runtime

At present, the algorithm runs in approximately 30 secondsper frame pair on the cadaver dataset. That is 0.13 secondsfor feature processing, 18.48 seconds for motion estimation,0.27 seconds for reconstruction, 8.78 seconds for Video/CTRegistration and 6.18 seconds for tracking. This was animprovement over the initial implementation that ran in

−6 −4 −2 0 2 4 601234567

Sensitivity of Initialization − In−plane

Perturbation Magnitude (mm)

TRE

(mm

)

*

**

**

**

* **

**Tracker−based

Video−based

(a)

−6 −4 −2 0 2 4 60

1

2

3

4Sensitivity of Initialization − Out of Plane

Perturbation Magnitude (mm)

TRE

(mm

)

* * *

Tracker−basedVideo−based

(b)

Fig. 14: A comparison of the Optotrak location TRE within-plane (a) and out-of-plane (b) perturbation and the video-based registration resulting TRE. A single star ( * ) indicatesp < 0.0024, α = 0.05/21 and no stars indicate no significantdifference (p > 0.0024, α = 0.05/21).

approximately two minutes per frame pair. We focused onaccuracy and robustness instead of speed in initial work. Thismethodology is common to vision literature. With furtherengineering, similar methods have been accelerated [43] andour algorithm could potentially be implemented as a real-timesystem suitable for clinical use. We realized a 300x speed-upof the feature processing by moving this component to theGraphic Processing Unit (GPU) and we continue to investigateporting to a GPU. We see further improvement within reach bytaking advantage of the nature parallelization of the samplingin the ASKC method.

V. DISCUSSION AND CONCLUSION

Our results show that video-CT registration can provide sub-millimeter visualization of targets. These results confirm our

14

previous hypothesis [16] that the video-CT solution is wellbelow 1 mm TRE.

In the phantom study, we saw that the Optotrak clearlyoutperformed all of the other methods. This is because thephantom was entirely rigid and the aluminium screw targetswere segmented and presented little beam-hardening artifact.Such a perfect scenario is only possible under the idealizedcase the phantom presented but does not represent the surgicalscenario.

We have shown that 2 mm accuracy is sufficient to initializeour system, thus existing systems such as the StealthStationprovide sufficient accuracy to benefit from video-based reg-istration. In the cadaver study, the 27 gauge syringe needlespresented a segmentation challenge as well as a smaller targetto touch with a pointer. However, the needle targets are stillclear in the video, which explains the improvement of thevideo-based method in the cadaver study and demonstrates thevideo-based method as the best method for the cadaver study.We observed, in the first cadaver study, conditions in whichour method failed to register but we were able to correct forthese with minor changes to the camera settings.

We continue to investigate robust features and trackingmethods. Although SIFT features are capable of tracking, theydo not track well over the large translations or large illuminationchanges that are expected endoscopy. For a more accuratereconstruction a large translation is preferred. There has alsobeen recent work in the computer vision literature showingthat real-time dense reconstruction is possible [43]. However,the method is tested on images of an office environment withdiverse texture. It remains to be shown if the accuracy of themethod holds with endoscope images. Other tracking methodswe are investigating include direct image registration usingimage similarity to optimize the pose of the virtual rendering.

In addition to speed for clinical use there are other challengesincluding smoke, dust and bleeding. We are in the process ofreceiving institutional review board (IRB) approval to recordvideo and track the endoscope in clinical cases. This data willprove valuable in validating our method. To deal with theseschallenges, we are looking into ways of toggling video trackingfor only video that is clear. To generalize to other areas ofendoscopy, the basis for our method would need to be updatedto soften our rigid assumption to a semi-rigid or deformablemodel or utilize intraoperative CT [44].

We also acknowledge that our algorithm does require thatthere be enough 3D structure and 2D texture in the sceneto register and to track. If the endoscope is pointed at a flatsurface or in a textureless region such as a surface covereduniformly with blood, our algorithm would not perform well asis. Additionally, if the camera is imaging a plane or symmetrictube, the registration will fail and the camera would needto be moved to a location imaging sufficient 3D structurerequired to solve the registration. Although, enough motion(approximately 2 mm) is required for an initial registration,afterwards the camera is still able to be tracked from the 2D-3Dcorrespondences, even during periods of small image-to-imagemotion or no motion. However, our algorithm could be used forlocal registration enhancement and would therefore add higheraccuracy capability to existing tracking systems. We envision

a visual re-registration feature that would offer surgeons theoption to have higher accuracy for the current scene.

ACKNOWLEDGMENTS

A preliminary version of this work was presented at amain track of MICCAI 2009 [16], CVPR 2008 [22] and SPIEMedical Imaging 2009 [21]. We thank Jeffrey H. Siewerdsenfor the use of the TREK software platform for the evaluationof the data. Special thanks to Mariela Zeledón for her kindeditorial comments. Finally, we thank the reviewers for theirhelpful suggestions that helped improve the paper significantly.

REFERENCES[1] S. S. Nasseri, J. L. Kasperbauer, S. E. Strome, T. V. McCaffrey, J. L.

Atkinson, and F. B. Meyer, “Endoscopic transnasal pituitary surgery:Report on 180 cases,” American Journal of Rhinology, vol. 15, no. 4,pp. 281–287(7), July-August 2001.

[2] F. Chassat and S. Lavalle, “Experimental protocol of accuracy evaluationof 6-d localizers for computer-integrated surgery: Application to fouroptical localizers,” in Medical Image Computing and Computer-AssistedInterventation MICCAI98, 1998, pp. 277 – 284. [Online]. Available:http://www.springerlink.com/content/x6632056v83043p2

[3] R. Shahidi, M. Bax, C. Maurer Jr, J. Johnson, E. Wilkinson, B. Wang,J. West, M. Citardi, K. Manwaring, and R. Khadem, “Implementation,calibration and accuracy testing of an image-enhanced endoscopy system,”Medical Imaging, IEEE Transactions on, vol. 21, no. 12, pp. 1524–1535,Dec 2002.

[4] R. Lapeer, M. S. Chen, G. Gonzalez, A. Linney, and G. Alusi, “Image-enhanced surgical navigation for endoscopic sinus surgery: evaluatingcalibration, registration and tracking,” The International Journal ofMedical Robotics and Computer Assisted Surgery, vol. 4, no. 1, pp.32–45, 2008. [Online]. Available: http://dx.doi.org/10.1002/rcs.175

[5] W. Konen, S. Tombrock, and M. Scholz, “Robust registration proceduresfor endoscopic imaging,” Medical Image Analysis, vol. 11, no. 6, pp. 526– 539, 2007. [Online]. Available: http://www.sciencedirect.com/science/article/B6W6Y-4NSX091-2/2/00030b4ca5f6c77b85e79426e7163fae

[6] C. Winne, M. Khan, F. Stopp, E. Jank, and E. Keeve, “Overlayvisualization in endoscopic ent surgery,” International Journalof Computer Assisted Radiology and Surgery, pp. 1–6, 2010,10.1007/s11548-010-0507-7. [Online]. Available: http://dx.doi.org/10.1007/s11548-010-0507-7

[7] F. Schulze, K. Bhler, A. Neubauer, A. Kanitsar, L. Holton,and S. Wolfsberger, “Intra-operative virtual endoscopy for imageguided endonasal transsphenoidal pituitary surgery,” InternationalJournal of Computer Assisted Radiology and Surgery, vol. 5,pp. 143–154, 2010, 10.1007/s11548-009-0397-8. [Online]. Available:http://dx.doi.org/10.1007/s11548-009-0397-8

[8] M. J. Daly, H. Chan, E. Prisman, A. Vescan, S. Nithiananthan,J. Qiu, R. Weersink, J. C. Irish, and J. H. Siewerdsen, “Fusion ofintraoperative cone-beam ct and endoscopic video for image-guidedprocedures,” in Medical Imaging 2010: Visualization, Image-GuidedProcedures, and Modeling, K. H. Wong and M. I. Miga, Eds.,vol. 7625, no. 1. SPIE, 2010, p. 762503. [Online]. Available:http://link.aip.org/link/?PSI/7625/762503/1

[9] J. P. Helferty, E. A. Hoffman, G. McLennan, and W. E. Higgins, “Ct-video registration accuracy for virtual guidance of bronchoscopy,” inSPIE Medical Imaging 2004: Physiology, Function, and Structure fromMedical Images, A. Amini and A. Manduca, Eds., vol. 5369. SPIEProceedings, 2004.

[10] D. Burschka, M. Li, M. Ishii, R. H. Taylor, and G. D. Hager,“Scale-invariant registration of monocular endoscopic images to ct-scansfor sinus surgery,” Medical Image Analysis, vol. 9, pp. 413–426, October2005. [Online]. Available: http://www.sciencedirect.com/science/article/B6W6Y-4GKWJ81-2/2/7eb32edc03ca267a0ff7c945a1766e5f

[11] B. Allain, M. Hu, L. Lovat, R. Cook, S. Ourselin, and D. Hawkes,“Biopsy site re-localisation based on the computation of epipolarlines from two previous endoscopic images,” in Medical ImageComputing and Computer-Assisted Intervention MICCAI 2009, ser.Lecture Notes in Computer Science, G.-Z. Yang, D. Hawkes,D. Rueckert, A. Noble, and C. Taylor, Eds. Springer Berlin/ Heidelberg, 2009, vol. 5761, pp. 491–498. [Online]. Available:http://dx.doi.org/10.1007/978-3-642-04268-3 61

15

[12] E. Delponte, F. Isgr, F. Odone, and A. Verri, “Svd-matching using siftfeatures,” in Proceedings of the of the International Conference on Vision,Video and Graphics, July 2005, pp. 125–132.

[13] H. Wang, D. Mirota, and G. D. Hager, “A generalized kernel consensusbased robust estimator,” IEEE Transactions on Pattern Analysis andMachine Intelligence, vol. 32, no. 1, pp. 178–184, 2010. [Online].Available: http://www.cs.jhu.edu/∼hwang/papers/tpami09.pdf

[14] K. Mori, D. Deguchi, J. Sugiyama, Y. Suenaga, J. Toriwaki, C. R. M.Jr., H. Takabatake, and H. Natori, “Tracking of a bronchoscope usingepipolar geometry analysis and intensity-based image registration of realand virtual endoscopic images,” Medical Image Analysis, vol. 6, pp. 321–336, 2002. [Online]. Available: http://www.sciencedirect.com/science/article/B6W6Y-46WFRXX-2/2/db4f20346eb50ae2f29908fe88ee28f7

[15] X. Luó, M. Feuerstein, D. Deguchi, T. Kitasaka, H. Takabatake,and K. Mori, “Development and comparison of new hybrid motiontracking for bronchoscopic navigation,” Medical Image Analysis, vol.In Press, Corrected Proof, pp. –, 2010, in Press. [Online]. Available:http://www.sciencedirect.com/science/article/pii/S1361841510001271

[16] D. Mirota, R. H. Taylor, M. Ishii, and G. D. Hager, “Direct endoscopicvideo registration for sinus surgery,” in Medical Imaging 2009: Visu-alization, Image-guided Procedures and Modeling. Proceedings of theSPIE, vol. 7261, February 2009, pp. 72 612K–1 – 72 612K–8.

[17] D. G. Lowe, “Distinctive image features from scale-invariantkeypoints,” International Journal of Computer Vision, vol. 60, pp. 91–110, 2004, 10.1023/B:VISI.0000029664.99615.94. [Online]. Available:citeseer.ist.psu.edu/lowe04distinctive.html

[18] A. Vedaldi, “An implementation of sift detector and descriptor,”Available online at http://vision.ucla.edu/∼vedaldi/code/sift/sift.html,2006, last accessed Mar. 3, 2009. [Online]. Available: http://vision.ucla.edu/∼vedaldi/code/sift/sift.html

[19] P. Mountney, B. Lo, S. Thiemjarus, D. Stoyanov, and G. Zhong-Yang,“A probabilistic framework for tracking deformable soft tissue inminimally invasive surgery,” in Medical Image Computing andComputer-Assisted Intervention MICCAI 2007, ser. Lecture Notesin Computer Science, N. Ayache, S. Ourselin, and A. Maeder, Eds.Springer Berlin / Heidelberg, 2007, vol. 4792, pp. 34–41. [Online].Available: http://dx.doi.org/10.1007/978-3-540-75759-7 5

[20] G. L. Scott and H. C. Longuet-Higgins, “An algorithm for associatingthe features of two images,” Proceedings: Biological Sciences,vol. 244, no. 1309, pp. pp. 21–26, 1991. [Online]. Available:http://www.jstor.org/stable/76644

[21] D. Mirota, H. Wang, R. H. Taylor, M. Ishii, and G. D. Hager, “Towardvideo-based navigation for endoscopic endonasal skull base surgery,” inMedical Image Computing and Computer-Assisted Intervention – MICCAI2009, ser. Lecture Notes in Computer Science, G.-Z. Yang, D. Hawkes,D. Rueckert, A. Noble, and C. Taylor, Eds., vol. 5761. Springer, 2009,pp. 91–99.

[22] H. Wang, D. Mirota, M. Ishii, and G. D. Hager, “Robust motion estimationand structure recovery from endoscopic image sequences with an adaptivescale kernel consensus estimator,” in Computer Vision and PatternRecognition, 2008. CVPR 2008. IEEE Conference on, June 2008, pp.1–7.

[23] R. I. Hartley and A. Zisserman, Multiple View Geometry in ComputerVision, 2nd ed. Cambridge University Press, ISBN: 0521540518, 2004.

[24] D. Nister, “An efficient solution to the five-point relative pose problem,”Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 26,no. 6, pp. 756–770, June 2004.

[25] Y. Ma, S. Soatto, J. Kosecka, and S. Sastry, An Invitation to 3-D Vision.New York: Springer Verlag, 2004, pp. 107–227.

[26] J.-Y. Bouget. (2010, July) The matlab camera calibration toolkit.Available online at http://www.vision.caltech.edu/bouguetj/calib\ doc/.Last accessed Oct. 15, 2010. [Online]. Available: http://www.vision.caltech.edu/bouguetj/calib doc/

[27] D. Abretske, D. Mirota, G. Hager, and M. Ishii, “Intelligent frame selec-tion for anatomic reconstruction from endoscopic video,” in Applicationsof Computer Vision (WACV), 2009 Workshop on, dec. 2009, pp. 1 –5.

[28] K.-M. Lee, P. Meer, and R.-H. Park, “Robust adaptive segmentationof range images,” Pattern Analysis and Machine Intelligence, IEEETransactions on, vol. 20, no. 2, pp. 200 –205, feb 1998.

[29] T. S. Newman and H. Yi, “A survey of the marching cubesalgorithm,” Computers & Graphics, vol. 30, no. 5, pp. 854 – 879,2006. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0097849306001336

[30] D. Chetverikov, D. Svirko, D. Stepanov, and P. Krsek, “The trimmediterative closest point algorithm,” Pattern Recognition, 2002. Proceedings.16th International Conference on, vol. 3, pp. 545–548 vol.3, 2002.

[31] S. Du, N. Zheng, S. Ying, Q. You, and Y. Wu, “An extension of the icpalgorithm considering scale factor,” Image Processing, 2007. ICIP 2007.IEEE International Conference on, vol. 5, pp. 193–196, 2007.

[32] P. Besl and H. McKay, “A method for registration of 3-d shapes,” PatternAnalysis and Machine Intelligence, IEEE Transactions on, vol. 14, no. 2,pp. 239–256, Feb. 1992.

[33] E. E. Catmull, “A subdivision algorithm for computer display of curvedsurfaces.” Ph.D. dissertation, The University of Utah, 1974.

[34] C.-P. Lu, G. Hager, and E. Mjolsness, “Fast and globally convergentpose estimation from video images,” Pattern Analysis and MachineIntelligence, IEEE Transactions on, vol. 22, no. 6, pp. 610–622, June2000.

[35] A. Uneri, S. Schafer, D. Mirota, S. Nithiananthan, Y. Otake, R. Taylor, andJ. Siewerdsen, “Trek: an integrated system architecture for intraoperativecone-beam ct-guided surgery,” International Journal of Computer AssistedRadiology and Surgery, pp. 1–15, 2011, in Press - (accepted) Jun 9,2011. [Online]. Available: http://dx.doi.org/10.1007/s11548-011-0636-7

[36] Python. Python Software Foundation. Last accessed on March, 8th 2011.[Online]. Available: http://www.python.org/

[37] A. Schmolck and V. Rathod. (2009, 10) mlabwrap. Website. Last accessedon March, 8 2011. [Online]. Available: http://mlabwrap.sourceforge.net/

[38] S. Pieper, B. Lorensen, W. Schroeder, and R. Kikinis, “The na-mic kit:Itk, vtk, pipelines, grids and 3d slicer as an open platform for the medicalimage computing community,” in Biomedical Imaging: Nano to Macro,2006. 3rd IEEE International Symposium on, april 2006, pp. 698 –701.

[39] A. Deguet, R. Kumar, R. Taylor, and P. Kazanzides, “The cisst librariesfor computer assisted intervention systems,” in The MIDAS Journal -Systems and Architectures for Computer Assisted Interventions (MICCAI2008 Workshop), 2008, https://trac.lcsr.jhu.edu/cisst/. [Online]. Available:http://www.midasjournal.org/browse/publication/623

[40] K. H. Strobl, W. Sepp, S. Fuchs, C. Paredes, and K. Arbter. (2010)DLR CalDe and DLR CalLab. Available online at http://www.robotic.dlr.de/callab/. Institute of Robotics and Mechatronics, German AerospaceCenter (DLR). Oberpfaffenhofen, Germany. Last accessed Oct. 15, 2010.[Online]. Available: http://www.robotic.dlr.de/callab/

[41] J. Fitzpatrick and J. West, “The distribution of target registration error inrigid-body point-based registration,” Medical Imaging, IEEE Transactionson, vol. 20, no. 9, pp. 917–927, Sept. 2001.

[42] D. J. Mirota, A. Uneri, S. Schafer, S. Nithiananthan, D. D. Reh,G. L. Gallia, R. H. Taylor, G. D. Hager, and J. H. Siewerdsen,“High-accuracy 3d image-based registration of endoscopic videoto c-arm cone-beam ct for image-guided skull base surgery,” inMedical Imaging 2011: Visualization, Image-Guided Procedures,and Modeling, K. H. Wong and D. R. H. III, Eds., vol. 7964,no. 1. SPIE, 2011, pp. 79 640J–1 – 79 640J–10. [Online]. Available:http://link.aip.org/link/?PSI/7964/79640J/1

[43] R. Newcombe and A. Davison, “Live dense reconstruction with a singlemoving camera,” in Computer Vision and Pattern Recognition (CVPR),2010 IEEE Conference on, june 2010, pp. 1498 –1505.

[44] J. H. Siewerdsen, M. J. Daly, H. Chan, S. Nithiananthan, N. Hamming,K. K. Brock, and J. C. Irish, “High-performance intraoperativecone-beam ct on a mobile c-arm: an integrated system for guidance ofhead and neck surgery,” in SPIE Medical Imaging, M. I. Miga and K. H.Wong, Eds., vol. 7261. SPIE, 2009, p. 72610J. [Online]. Available:http://link.aip.org/link/?PSI/7261/72610J/1