problem 2 rapid modeling of internal structures of ... · industrial problem‐solving workshop on...
TRANSCRIPT
Industrial Problem‐Solving Workshop on Medical Imaging
Problem # 2Rapid Modeling of Internal Structures of p g
Deformable Organs (i.e. Liver)
Edward Xishi Huang and James DrakeThe Hospital for Sick Children
Problem definition
Accurate estimation of deformation of the soft organ’s internal structures between two images acquired at differentinternal structures between two images acquired at different conditions
What information are used?! Point landmarks [1] Vessel segments and curves [1]
Surface information
Problem definition
Application: Tumor treatment using high‐intensity focused ultrasound (HIFU) thermal proceduresultrasound (HIFU) thermal procedures.
• Real‐time image guidance is required for targeting. • Intraoperative imaging systems:
– Low resolution and quality images– Long acquisition time– Compatibility with other equipments in the operation room.
Solution: Updating preoperative treatment plans and physical deformation models based on intraoperatively acquired images.gThe idea: Employing boundary conditions (i.e. curves of vessels and bifurcations of vessels, landmarks, and surface information ) to constrain the solution of the deformableinformation ) to constrain the solution of the deformable model of organ.
Problem definition
Registering internal structures and surface information of the two set of images acquired from the soft organtwo set of images acquired from the soft organ.
Image Registration
Given a reference image R and a template image T, find aGiven a reference image R and a template image T, find a reasonable transformation y, so that the transformed image T[y]is similar to R [2].
Problem 1
Modeling internal structures in 2D images using landmarks and vessel segments with known end pointsand vessel segments with known end points
9
10
C 9
10
6
7
8
6
7
8
C1
3
4
5
D
B
4
5
B1
A
( )S x( )S x′ ′
1
2
3
A
1
2
3
D1
A1
0 2 4 6 8 100
0 2 4 6 8 100
Problem 1
Solution
L d k ( ) ( )d f l′ ′Landmarks:
Vessel Curves:
( ) ( ), and , for 1,2,...,l l l lx x x y l n′ ′ =
( ) and ( )y S x y S x′ ′ ′= =Vessel Curves:
Displacement Function:
( ) and ( )y S x y S x
( )( , ) , ( , )u x y v x yp
Landmarks:
( )( ) ( )y y
( ) ( ), ( , ), ( , )l l l l l l l lx y x u x y y v x y′ ′ ⇐ + +
Any point on the curve: ( ) ( ), ( , ), ( , )x y x u x y y v x y′ ′ ⇐ + +
Problem 1
Solution
Th f i b i i i d f l d k hiThe cost function to be minimized for landmarks matching:
( ) ( ){ }2 21( , ) ( , ) ( , )
n
l l l l l l l lJ u v x x u x y y y v x y′ ′= − − + − −∑
The cost function to be minimized for curve matching:
{ }1l=
( ) ( ) ( )( )( ) ( ) ( ) ( )x S x x u x S x S x v x S x⇒ + +This point corresponds to
( ) ( ) ( )( ), ( ) , ( ) , ( ) , ( )x S x x u x S x S x v x S x⇒ + +
( ) ( ) ( )( )( ), ( ) , ( ) , , ( )x S x x u x S x S x u x S x′⇒ + +( ) ( ) ( )( )( )( )( ) ( )
2
1
2
2 ( , ) , ( ) ( ) , ( )x
x
J u v S x u x S x S x v x S x dx′= + − −∫
Problem 1
Solution
Th l f i b i i i d f l d k dThe total cost function to be minimized for landmarks and curve matching:
1 1 2 2( , ) ( , ) ( , )J u v w J u v w J u v= +
Having the variational model in optimization model form:
0 ( , )N
i iu Ax By x a x yϕ= + + +∑01
01
( )
( , )
i iiN
i ii
y y
v Cx Dy y b x y
ϕ
ϕ
=
=
= + + +
∑
∑are radial basis functions.
The cost function in terms of unknown parameters
( 1,2,..., )i i Nϕ =
( )0 0 1 1min , , , , , , ,..., , ,...,N NJ A B C D x y a a b b
Problem 2
Modeling internal structures in 2D images using landmarks and vessel segments with one end pointand vessel segments with one end point.
9
B8
9
( )S x
6
7
8
6
7
8
( , )e ex y( )′ ′
( )S x′ ′
4
5
3
4
5
B1
1 1( , )x y′ ′
( , )e ex y′ ′
1 2 3 4 5 6 7 8 92
3A
1A A→1 2 3 4 5 6 7 8 9
2
3A11 1( , )x y
1 1( , )y
1
?!A AB→→
Problem 2
Solution
i k i b f d d d b dd d( ) is unknown point to be found and needs to be added to the optimization function as follows:
( , )e ex y
( )( ) ( )( )
( )( ) ( )
2 22
2
( , , ) , ( ) , ( )
( ) ( ) ( )
e e e e e e e e e
xe
J x u v x x u x S x y y v x S x
S x u x S x S x v x S x dx
′ ′= − − + − −
′+ + − −∫ ( )( ) ( )1
, ( ) ( ) , ( )x
S x u x S x S x v x S x dx+ +∫
Problem 3
Modeling internal structures in 3D image
L d k Th 2D iLandmarks: The same as 2D images
( ) ( ) ( ){ }1
2 2 2
( , , )n
J u v w =
∑
Curves:
( ) ( ) ( ){ }2 2 2
1( , , ) ( , , ( , ,l l l l l l l l l l l l l l l
lx x u x y z y y v x y z z z v x y z
=
′ ′ ′− − + − − + − −∑
Parametric cubic representation: piecewise spline3 2( ) x x x xx t a t b t c t d= + + +3 2
3 2
( )
( )
( )
x x x x
y y y y
z z z z
y t a t b t c t d
z t a t b t c t d
= + + +
= + + +
Problem 4
Registration based on surface matching of the object in two set of imagesset of images
( ), , ( , )x y S x y ( ), , ( , )x y S x y′ ′ ′ ′ ′
( )( , , ), ( , , ), ( , , )u x y z v x y z w x y z( )( , , ), ( , , ), ( , , )u x y z v x y z w x y z
Problem 4
The cost function to be minimized:
( ), , ( , )x y S x y ( ), , ( , )x y S x y′ ′ ′ ′ ′
( ) ( )( ) ( )2
( , , )
( ) ( ) ( ) ( )
J u v w
S S S S S d d
=
′∫Another solution: deformable surface modeling using active
conto rs [3]
( ) ( )( ) ( ), , ( , ) , , , ( , ) ( , ) , , ( , )S x u x y S x y y v x y S x y S x y w x y S x y dx dyΩ
′ + + − −∫
contours [3]
Impact
FEM‐based image registration.
I i b d [4]Intensity based [4]
[ ]( ) 1min ( ) , ;2
TJ u D T u R u Kuα α += + ∈ℜ[ ]( )2
[ ]( ),( ) 0D T u RJ u Ku
u uα
∂∂= + =
∂ ∂u u∂ ∂
[ ]( ),1( )D T u R
Ku f u∂
= = −∂
‐ Multimodality
uα ∂
‐ Local minima
Impact
FEM‐based image registration.
F b dFeature based
Internal structures and surface information of the deformable organ provides necessary boundary condition to solve theorgan provides necessary boundary condition to solve the FEM‐based deformation equation.
Example: Linear finite element model of the brain shift and deformation [3].
Impact
FEM‐based image registration.
References
[1] Dietlind Zühlke, Sven Arnold, Gernoth Grunst, Peter Wißkirchen, “Intra‐interventional registration of 3D ultrasound to models of the vascularinterventional registration of 3D ultrasound to models of the vascular system of the liver” GMS CURAC 2007, Vol. 2(1)
[2] J. Modersitzki, Numerical Methods for Image Registration. New York:Oxford, 2004.o O o d, 00
[3] M. Ferrant, A. Nabavi, B. Macq, F. Jolesz, R. Kikinis, and S. Warfield, “Registration of 3D Intraoperative MR Images of the Brain Using a Finite‐Element Biomechanical Model,” IEEE Trans. on Medical Imaging, vol. 20, , g g, ,no. 12, pp. 1384–1397, 2001.
[4] B. Marami, S. Sirouspour, and D. Capson, “Model‐Based Deformable Registration of Preoperative 3D to Intraoperative Low‐Resolution 3D and g p p2D Sequences of MR Images,” in MICCAI 2011, pp. 460–467, 2011.
Group Members
Edward Xishi Huang
C Sean BohunC. Sean Bohun
Tian Chen
Jamil Jabbour
Ken Jackson
Iain Moyles
Cartihk Sharma
Yongji Tan
Bahram Marami
Blah
Motivation
Modelling a three-dimensional image of the liver iscomputationally expensive (and has challenging physics)
Image data arrives in several two-dimensional cross sections(slices) as the body is scanned
Each slice is identified by unique markers (landmarks andcurves)
If all of the markers from one slice map to all be on atransformed slice then we can consider a series oftwo-dimensional transformations
Blah
Setup
LandmarksThe N distinguishing points on image
CurvesThe κ distinguishing curves (vessels) on image
Assume a transformation from“pre-image” (X ,Y ) to“post-image” (X , Y ) via a transformation (X , Y )
X = X +∑j
ajφj(X ,Y ),
Y = Y +∑j
bjφj(X ,Y ),
φj are radial basis functions (RBF)
φj(X ,Y ) = exp
(−
(X − Xj)2 + (Y − Yj)
2
σ
)(Xj ,Yj) are the L allocation points to form a basis set (L ≤ N)
Blah
Setup
For curves we assume there exists a mapping for curve pointson the pre-image (xx , yy) given by yy = S(xx)
Similarly there exists a map yy = S(xx) for the curve pointson the post-image (xx , yy)
Often the form of S and S will be through an interpolation(cubic splines)
The curves have the requirement that the endpoints arelandmarks
Blah
Pre Image
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Student Version of MATLAB
Blah
Post Image
187 187.5 188 188.5 189 189.5 190 190.5256
256.5
257
257.5
258
258.5
259
259.5
260
Student Version of MATLAB
Blah
Optimization
We wish to determine the coefficients for the RBF thattransforms the pre-image into the post-image
We determine these by minimizing an error function composedof matching landmarks and curves
Therefore we consider1 Landmark Error
We wish all terms of the form X − X to be small so that thelandmarks are close
2 Curve Error
We wish the curve points to be close as well i.e. xx − xx andS(xx) − S(xx) are smallThe predicted curve S(xx) = S(xx) +
∑j bjφj(xx , S(xx))
Blah
Solution
187 187.5 188 188.5 189 189.5 190 190.5256
256.5
257
257.5
258
258.5
259
259.5
260
189.2 189.4 189.6 189.8 190 190.2
258.6
258.8
259
259.2
259.4
259.6
259.8
Blah
Oops...
186 186.5 187 187.5 188 188.5 189 189.5 190 190.5254
255
256
257
258
259
260
261
188.8 189 189.2 189.4 189.6 189.8 190 190.2 190.4
258
258.5
259
259.5
260
260.5
Blah
Constrained Optimization
There are many local minima to this problem with largeenergies
In fact, there are many local minima with small energies thatdon’t represent the “true” solution (lack of global minimizer?)
Idea: Build in a constraint for each curve that forces the areaunder the post-image curve to match that of the predictedcurve ∫ xx2
xx1
(S(xx) − S(xx) −∑j
bjφj) dxx = 0
Blah
Solution
187 187.5 188 188.5 189 189.5 190 190.5256
256.5
257
257.5
258
258.5
259
259.5
260
189.4 189.5 189.6 189.7 189.8 189.9 190 190.1 190.2
258.6
258.8
259
259.2
259.4
259.6
259.8
Blah
Conclusions
Created image data
Performed an optimization to recover RBF coefficients thattransform pre-image to post-image
Included integral constraints to reduce set of minimizers
Future Work
Apply to real landmark data
Consider weightings carefully
Center of mass type penalty systemVoronoi diagrams
Allow for curve endpoints to move