1/20 obtaining shape from scanning electron microscope using hopfield neural network yuji iwahori 1,...
TRANSCRIPT
1/20
Obtaining Shape from Scanning Electron Microscope Using Hopfield Neural Network
Yuji Iwahori1, Haruki Kawanaka1, Shinji Fukui2 and Kenji Funahashi11 Nagoya Institute of Technology, Japan
2 Aichi University of Education, Japan
2/20
Introduction
Shape from Scanning Electron Microscope (SEM) images is the recent topic in computer vision.The position of a light source and a viewing point are the same under the orthographic projection. The object stand is rotated to some extent through the observation.
Only these conditions can be used to recover the object shape.
2D Image of SEM
Recovering 3D Shape
3/20
Previous Approaches (1)
Photometric Stereo
Estimation using the temporal color space
use multiple images under the different light source directions.
Linear Shape from Shading
Photometric Motionthe position of viewing point (camera) and light source should be widely located
4/20
Previous Approaches (2)
Shape from Occluding Boundariesis limited to a simply convex closed curved surface
Shape from Silhouetteuses multiple images through 360 degree rotation,
is also unavailable to object with local concave shape
Surface Reflectance and Shape from Images Using 90 degree rotation to get the feature points
However the rotation angle is limited to SEM
5/20
New Proposed Approach
Uses optimization with two images observed through the rotation of the object stand
1. The appropriate initial vector is determined using the Radial Basis Function neural network (RBF-NN) from two images during rotation.
2. The optimization is introduced using the Hopfield like neural network (HF-NN).
6/20
Characteristics of SEM Image (1)
Orthographic projection
Rotation angle
Reflectance property
i : incident angle, < 70°
s ≈ 0.5
R(i) is normalized to the range of 0 and 1.
3030
)1(cos
1)( i
ssiR
7/20
Characteristics of SEM Image (2)
z = F(x, y)F : height distribution
p, q : gradient parameters l = (0, 0, 1) : light source direction : surface normal
cos i = n ・ l = nz … (3)
From Eq.(1)(2)(3),
y
zq
x
zp
,
)2(
1
1,,
),,(
22
qp
qp
nnn zyxn
1),( 22 qpssqpR
Cross Section of Reflectance Map(q=0)
8/20
Rotation Axis on Object Stand
Under the orthographic projection the gradient of the rotation axis is the sam
e for both images observed during rotation.
ex.
9/20
Estimation of Rotation Axis
1. Assume A and B move A’ and B’ during the rotation
2. Set A and A’ be the same pixel
3. Then rotation axis is determined so that it becomes perpendicular to the line BB’ and passes through the point A.
10/20
Shape Recovery from Two Images Using Hopfield Neural Network
Hopfield Neural Network (HF-NN) the mutual connection network the connection between the ne
urons are the symmetric HF-NN can be applied to solve t
he optimization problem of the energy function
m1
m2
m3
11/20
Energy Function to be Minimized
C1, C2, C3 : the regularization parameters
D : the target region of the object
E1 : the smoothness constraint
E2 : the error of the observed image brightness I(x,y) and the reflectance map R(p, q)
E3 : the error of the geometric relation for z and (p, q)
D
D
D
dxdyqy
zp
x
zE
dxdyqpRyxIE
dxdyy
q
x
q
y
p
x
pE
ECECECE
22
3
22
2222
1
332211
),(),(
(p,q,z): unknown variables
12/20
Initial Vector for Optimization (1)
Radial Basis Function Neural Network (RBF-NN) is introduced to obtain the approximation of gradient p, qAssume the same pixel (x, y) during the rotation.
The integration of along x direction results in the height distribution.
RBF
NN
I1(x, y)
I2(x, y)
nx
nz
z
x
n
np
13/20
Initial Vector for Optimization (2)
How to make dataset of RBF-NN A sphere is used to make I1 and I2 using R(p,
q), where, R is
since a sphere has the whole combination of the surface gradient.How to use learned RBF-NN
The corresponding point of the target object is assumed to be the same during the rotation.
1),( 22 qpssqpR
14/20
Updating Equation using HF-NN
The equation is iteratively used to optimize the energy function, that is, each partial difference becomes 0.
p
qpRqpRyxIC
p
E
y
p
x
pC
p
E
p
E
p
E
p
E
p
E
t
p
y
q
y
z
x
p
x
zC
z
Ez
E
z
E
t
z
),(),(),(2
2
2
22
2
2
2
2
11
321
2
2
2
2
33
3
qy
zC
q
E
q
qpRqpRyxIC
q
E
y
q
x
qC
q
E
q
E
q
E
q
E
q
E
t
q
px
zC
p
E
33
22
2
2
2
2
11
321
33
2
),(),(),(2
2
2
15/20
Iteration for Optimization
The optimization is applied to each of two images repeatedly. The height z’with the rotation angle is given byGradient are also calculated from the height repeatedly during rotation.C1 is gradually reduces
E1 : the smoothness constraint
Optimization is terminated the value of energy function converges in comparison with that of one step before.
cos),(sin),( yxzxyxz
332211 ECECECE
16/20
Experiments (synthesis image)
Rotation angle is 10°Image size is 64×64 pixels Rotation axis is along the center of the image.RBF-NN
Learning Data : 2000 Learning Epoch : 15
MSE 1.8961Maximum Height 10.37Theoretical Height Initial Height Recovered Height
Input Images
17/20
Experiments (SEM image)
Rotation angle taken is 10 °Rotation axis is set from the known feature points A and B
MSE 3.8926Theoretical Depth 13.1031
Theoretical Height Initial Height Recovered HeightRelaxation Method
Input Images
18/20
Experiments (SEM image)
Rotation angle taken is 10 °Rotation axis is set from the known feature points A and B
Input Images
Initial Height Recovered Height
19/20
Conclusion
A new method is proposed to recover the shape from SEM images.
HF-NN is introduced to solve the optimization problem. The energy function is formulated from two image during rotation.
The initial vector is obtained using RBF-NN.
20/20
Further works
Getting more accurate result using more images
Treatment of the inter-reflection
Thank you