multi-sensor imagery processing for military application · 2013. 1. 24. · multi-sensor imagery...
TRANSCRIPT
Multi-Sensor Imagery Processing
for Military Application
2013. 1. 18. (fri) 10:40~11:10
Sang-Hee Kim, Ph.D.
Project Manager in ISR center of ADD
Motivation Ⅰ
GEOINT Data Generation Ⅱ
Multi-sensor Imagery Matching Ⅳ
Conclusion Ⅴ
3-D Model Reconstruction Ⅲ
MOTIVATION
R&D
www.armyrecognition.com/News/2007/March/Milit...
Victory smiles upon
those who anticipate
the changes in the
character of war,
not upon those who
wait to adapt themselves a
fter the changes occur.
-- Italian Air Marshall Giulio Douhet
www.journal.dnd.ca/vo8/no4/robertso-eng.asp
Platforms Intelligence
(PGM) ≫ (ISR)
Platforms Intelligence
(PGM) ≈ (ISR)
SMART system
Smart Multi-INT Application for Relevant Targeting
– Multi-platform/multi-sensor imagery integration
– Image/Video Exploitation and Analysis
– Accurate 3-D Model Reconstruction and Management
– Real-time Sensor ATR
Minimization of off-target attack
– Performance enhancement of weapon system
– Accurate target intelligence and timely transmission
Complex targets
– Randomly shaped facility, natural occlusion or camouflage
표적의 복잡, 다양성
8/74
위협요소 (AAA/SAM 모델 및 화망)
GEOINT activities
Satellite-GRPS, Airborne-GRPS, UAV-GRPS – Ground Receiving and Processing System
– EO/IR(image and/or video), SAR and MTI
Image-based Point Positioning System – cf, Raindrop or CGS on DPPDB in USA
3-D Model Reconstruction and Matching – Geo-spatial data generation
– 2-D terrain feature and 3-D facility extraction
– Geometry-based and image-based matching
Multimedia Database for Threat Detection – Feature-based image analysis and retrieval
GEOINT Data Generation (DEM and Ortho-image)
Receiving Range
Antenna /Receiving Device
Satellites
SPOT-2
SPOT-4
Radarsat-1
2,000km
•Data Acquisition System
GRPS system
Image acquisition Stereo images3D modeling
Regular grid values Coordinate calculation Matching
Ortho-image (texture)Indirect rectificationDigital Elevation Model
left right
left right
matching
point
matching
point
Groundcoordinate
DEM
flat DEM
Image acquisition Stereo images3D modeling
Regular grid values Coordinate calculation Matching
Ortho-image (texture)Indirect rectificationDigital Elevation Model
left right
left right
matching
point
matching
point
Groundcoordinate
DEM
flat DEM
Rigorous Sensor Modeling (SPOT)
Equation
– GCP / Image
– EOP
Exterior Orientation Parameter
– Satellite Position: (XL, YL, ZL)
– Attitude: (L, L, L)
X
Y
Z
O
GCP(Xgcp,Ygcp,Zgcp)
(iR,jR)
(iL,jL)
EOP(XL,YL,ZL,L, L, L) EOP(XR,YR,ZR,R, R, R) X X X t X t X t
Y Y Y t Y t Y t
Z Z Z t Z t Z t
L 0 1 2
2
n
n
L 0 1 2
2
n
n
L 0 1 2
2
n
n
...
...
...
2nd polynomial
3 Dimensional Modeling
Strip mode
0 11 12 13
21 22 23
31 32 33
y
f
R
X X
Y Y
Z Z
R R R R
m m m
m m m
m m m
L
L
L
L
L L L L
, ( ) ( ) ( )
X X X t X t
Y Y Yt Y t
Z Z Z t Z t
t t
t t
t t
L
L
L
L
L
L
0 1 22
0 1 22
0 1 22
0 1 22
0 1 22
0 1 22
Collinearity (SPOT line L)
o Satellite orbit parameter
o equation
F fm X X m Y Y m Z Z
m X X m Y Y m Z Z
G y fm X X m Y Y m Z Z
m X X m Y Y m Z Z
i L i L i L
i L i L i L
ii L i L i L
i L i L i L
0 11 12 13
31 32 33
11 12 13
31 32 33
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
o Taylor series
FF F F F
XX
F
YY
F
ZZ
F F F F
XX
F
YY
F
ZZ
F F F F
00
00
00
00
00
00
0
11
11
11
11
11
11
22
22
22
X
XF
YY
F
ZZ
22
22
22 0
GG G G G
XX
G
YY
G
ZZ
G G G G
XX
G
YY
G
ZZ
G G G G
00
00
00
00
00
00
0
11
11
11
11
11
11
22
22
22
X
XG
YY
G
ZZ
22
22
22 0
o Observation equation construction
GCP
orbit
image
C
C
C
I
A
I
B
V
V
V
2
1
2
1 0
0
o Simplified equation
V B C
p n m p n m n m n m p n m( , ) ( , ) ( , ) ( , )2 18 3 1 2 18 3 18 3 18 3 1 2 18 3 1
RPC(Rational Polynomial Coefficients)
Developed for IKONOS by Space Imaging Inc.
WGS84, Geographic coordinate _
_
_
_
_
_
Latitude LAT OFFP
LAT SCALE
Longitude LONG OFFL
LONG SCALE
Height HEIGHT OFFH
HEIGHT SCALE
( , , )( , , )
( , , )
( , , )( , , )
( , , )
L
L
S
S
Num P L HY f Latitude Longitude Height
Den P L H
Num P L HX g Latitude Longitude Height
Den P L H
2 2
1 2 3 4 5 6 7 8 9
2 3 2 2 2 3
10 11 12 13 14 15 16
2 2 2 3
17 18 19 20
2 2
1 2 3 4 5 6 7 8 9
2 3 2 2 2
10 11 12 13 14 15
L
L
Num a a L a P a H a LP a LH a PH a L a P
a H a PLH a L a LP a LH a L P a P
a PH a L H a P H a H
Den b b L b P b H b LP b LH b PH b L b P
b H b PLH b L b LP b LH b L
3
16
2 2 2 3
17 18 19 20
P b P
b PH b L H b P H b H
2 2
1 2 3 4 5 6 7 8 9
2 3 2 2 2 3
10 11 12 13 14 15 16
2 2 2 3
17 18 19 20
2 2
1 2 3 4 5 6 7 8 9
2 3 2 2 2
10 11 12 13 14 15
S
S
Num c c L c P c H c LP c LH c PH c L c P
c H c PLH c L c LP c LH c L P c P
c PH c L H c P H c H
Den d d L d P d H d LP d LH d PH d L d P
d H d PLH d L d LP d LH d L
3
16
2 2 2 3
17 18 19 20
P d P
d PH d L H d P H d H
_ _
_ _
Line Y LINE SCALE LINE OFF
Sample X SAMP SCALE SAMP OFF
RPC fitting
Gene Dial, Jacek Grodecki, Block adjustment with rational polynomial camera models,
ACSM-ASPRS 2002 Annual Conference Proceedings, 2002.
( ) ( ) ( )
( ) ( ) ( )
( , , )
( , , )
( , , ) ( , , ) _ _
( ,
ij
ij
j j j
i k k k L
j j j
i k k k S
Line p p Latitude Longitude Height
Sample r r Latitude Longitude Height
p Latitude Longitude Height f Latitude Longitude Height LINE SCALE LINE OFF
r Latitude Lo
0
0
, ) ( , , ) _ _
S L
S L
ngitude Height g Latitude Longitude Height SAMPLE SCALE SAMPLE OFF
p a a Sample a Line
r b b Sample b Line
RPC : accuracies vary with satellites
Need to correct using minimal GCPs
RPC correction-1
pi=(x, y, lon, lat, alt)
Forward
form
Pi’=(x’, y’, lon, lat, alt)
Transformation Matrix – ≤ 2 points : Translation
– 3 points : Affine Transform
– ≥ 4 points : Homography
RPC correction-2
[ ] 3x3 Left image
[ ] 3x3 Right image
3-D
Reconstruction
RPC + GCP + DEM
Ortho-rectified Image
P=(X,Y,Z)
Z
Ortho image
(X,Y)
Interpolated pixel
p=(x,y) p’=(x’,y’)
correction
RPC Forward Form
DEM
input
process
Position Accuracy Evaluation
CMAS r0
90%
Circular MAS Linear MAS
measurement
Statistical analysis
Test points
LMAS r0
90%
Below c
Multi-resolution/Multi-layer Terrain
Modeling and controlling
Fused mesh of multi-resolution DEMs
Geo-consistent processing
higher terrain layers and base layers (DEM and texture)
3D model
Geo-specific (high resolution)
Geo-typical (low resolution)
3-D Model Reconstruction
GPS navigation
TERCOM
navigation
Image-based
navigation
GPS
interference
Mission start
Registered Optical and DEM
Parameterized Model Library
Site Model Visualization
Model Indexing
Elevation Surface and Histogram
Model F
it: O
ptim
ization
Automated 3-D Model Reconstruction(1/2)
Image Sequence
Feature Extraction/ Matching
Relating Image
Projective Reconstruction
Auto-Calibration
Dense Matching
3D Model Building
Automated 3-D Model Reconstruction(2/2)
Manual Extraction(1/2)
Vertex: 495
Face: 395
Vertex: 1226
Face: 886
Manual Extraction(2/2)
3D-METER(Model Extraction and Terrain Registration)
Depth map generation
Terrain & Feature
Integration
Ortho-rectificatio
n
3D Model Database
DTM, 3D Model, Texture Image, Model Hierarch, Orthophoto Etc…
3D Modeling
3D Height estimation
3D Model Editing
Texture Mapping
Stereo Matching
Terrain Feature and Boundary
추출 및 2D 모델링 Feature & Rooftop Extraction/Modification
Metadata
GCP
Image Data
RPC analysis
Image Processing
Single Image Dual Image Stereo Image
New project, GCP correction
ROI, Resolution merged, Ortho-Image
(1) Extraction of 2-D terrain feature
SNAKE(Active Contour Model) algorithm
N
i
idigisic pdEpcEpbEpaEE1
)()()()(
Internal energy External energy
cE
sE
“Continuity”
“Smoothness”
“Edgeness”
a, b, c, d : weights to control the curved shape
gE
dE “Clustering”
clustering term
to deal with
concave or highly
curved region
s
Pi
Edge attraction
Smoothness Continuity
Simple feature extraction (Road)
Complex feature(river): SNAKE+ISODATA+CPS
original ISODATA result Boundary energy
proposed snake snake
Comparison of # of control points
input control points
Snake control points
Control points
after CPS
building 14 168 80
road 9 343 86
river 120 1145 192
Extraction of 3-D Facilities
Rooftop morphology
Satellite and Sun position
Move Rooftop-shadow edge
Shadow direction estimation
Rooftop-shadow edge selection
Line evaluation
Shadow Region
Yes
No
)tan(/
)tan(/
sat
sun
HVL
HSL
Height calculation (H)
Facade line azimuth
su
Satellite azimuth
North Sun azimuth
Facade Normal
tsa
sun
san scan
Building
t
tsasan
tsusun
sansan
sunsun
sansun
KSH
VLS
SLS
SSS
90
90
)cos(
)cos(
S : Estimated Shadow length
))}tan(/)(cos())tan(/)/{(cos()sec( sansansunsunscantK
Consideration of Terrain elevation
Rooftop
Image plane
Terrain
S
p R1
R'1
hs
H
ht
S'
t
t
t
t
t
h
zDyDh
zDxDh
p
zD
ht
tD
h
y
x
p
./.
./.
.
Search the terrain patch in azimuth direction of Satellite
3-D Model Extraction(Facility)
Coordinate generation by image registration
3D models extracted and merged on terrain
QuickBird Aerial Image WorldView
Case2
04
Reference Target Data generation (Geometry+image) on sensor parameter
Accuracy evaluation of 3D model
sample : 10 facilities
1 3
Model #1
3D-METER
X : 246494.63
Y : 4063550.50
Z : 9.55
11.37 19.37
9.21
Model #3
30.66
10.74
5.06
3DM
X : 247068.84
Y : 4063560.25
Z : 12.23
11.37 19.37
9.21
Model Index Data
Points Model size
X Y Z WW W H
1
Measured 246495.132 4063550.269 12.029 20.115 12.168 8.907
3D-METER 246494.63 4063550.5 9.55 19.37 11.37 9.21
Difference
(m) 0.502 -0.231 2.479 0.745 0.798 -0.303
3
Measured 247069.465 4063560.056 13.66 30.638 10.386 6.46
3D-METER 247068.84 4063560.25 12.23 30.66 10.74 5.06
Difference
(m) 0.625 -0.194 1.43 -0.022 -0.354 1.4
Relative Error btw models (Test Site #3)
1
40.253
2
3
4 5
6
7
8
9
10 11
12
36.045 40.195
52.461
Site 1 (unit: m) Site 2 (unit: m) Site 3 (unit: m)
Absolute error
- X : 5.889
- Y : 1.375
-Z : 0.425
Size error : 0.283
Absolute error
- X : 7.121
- Y : 0.470
- Z : 0.195
Size error : 0.357
Relative error : 0.271
Absolute error
- X : 7.378
- Y : 0.986
- Z : 0.733
Size error : 0.570
Relative error : 0.694
Data : World view2, SRTM Level2
Relative Accuracy
Within 2 pixels in error !!
Multi-sensor Imagery Matching
49
Registration based on Multi-sensor imagery
Target
Detect/
Track
3D Model
Super-resolution
Initial match
Multi-sensor image registration
CCD IR Precise match
Image registration
Registration
);,(maxarg mfreg PIIP
PS
: Pixels in a fixed image
: Pixels in a moving image
If: Fixed image
Im: Moving image
S: Similarity measure
P: Parameter set of transformation model
Similarity measure
);( YXI
)(XH
),( YXH
)(YH
)|( YXH )|( XYH
),()()(
)|()(
)|()();(
YXHYHXH
XYHYH
YXHXHYXI
),(
)()();(
YXH
YHXHYXI
N
+1
+1
+1
+1
Fixed image intensity
Movin
g im
age inte
nsi
ty
<Joint histogram> EO image
IR image
Challenging Problems
LOST
LOCK - Target
CM : Flare
LOCK - Target
LOST
(1) Counter Measure
(2) Image Database
Image Retrieval Processing
Image Indexing & Storing
Input Image
Segmentation &
Feature Extraction Image Indexing Metadata
Query Image
Segmentation &
Feature Extraction Query Processing
Location data
Feature data
Annotation …
Meta
data
Meta
data
Meta
data
Image Database
Browsing Results
. . . . . . . .
Similarity
Function
Relevance
Feedback …
Threat Detection
(3) True Ortho-rectified Image
Traditional Ortho image
- Rectified from a DTM (floor)
- Slanted buildings
- Occulted areas
True Orthoimage
- Rectified from a DSM
- Straight buildings
(roofs correctly located)
- No occulted areas
Shadow area
Occluded area
(4) Fusion on Multiple Sensor Images
enhancement
Multi-sensor
Image
(real time)
fusion 3D model
Satellite image,
Lidar data
(off line)
Feature matching
3D model
(texture + terrain)
Satellite image Lidar
CCD IR or SAR
Image Fusion
Target recognition 3-D model information
(5) 3D Positioning with New Single Image
Dataset :
New single image, Satellite Stereo image Database(DPPDB, IBPPS, etc)
Procedure
DPPDB : Digital Point Positioning Data Base
IBPPS : Image Based Point Positioning System
Start
Calculate RPC coeff. of the single image
Observe Image Coord. of the Objects in the single image
and Sat. Stereo Images
Input Single Image
Observe Image Coord. of GCPs
Input Sat. Stereo Images and RPCs
Calculate GRD Coord. of GCPs
3D Geopositioning with multi-images RPC model
Finish
Observe Image Coord. of GCPs
RPC Sensor Modelingthe Single Image
(101) (102)
(103) (104)
(105)
(106)
(107)
(108)
(109)
H
(0,0,0)
i
iHi
( , , )i i i iGCP H
Sat. Stereo(Left)
( , )L Ll s
Sat. Stereo(Right)
( , )R Rl s
RRPCsLRPCs
RPCs
Single Image
( , )i il s(101)
(102)
(103)
(104)
(105)
(106)
(107)
(104)
Dataset :
New single image, Satellite Stereo image Database(DPPDB, IBPPS, etc)
Procedure
Start
Calculate RPC coeff. of the single image
Observe Image Coord. of the Objects in the single image
and Sat. Stereo Images
Input Single Image
Observe Image Coord. of GCPs
Input Sat. Stereo Images and RPCs
Calculate GRD Coord. of GCPs
3D Geopositioning with multi-images RPC model
Finish
Observe Image Coord. of GCPs
RPC Sensor Modelingthe Single Image
(101) (102)
(103) (104)
(105)
(106)
(107)
(108)
(109)
Ground
l
s
( , )L Ll s
l
s
( , )R Rl s
U,V,W
RRPCsLRPCs
l
s Single Image
( , )l s
RPCs
Sat. Stereo(Left) Sat. Stereo(Right)
(108)
(108) (109)
CONCLUSION
My ultimate research goal : SMART
GEOINT data generation using GRPS system
Target model reconstruction : 3D-METER
– One-stop integrated modeling software giving simple, efficient, accurate
(application-oriented system)
Multi-sensor imagery matching
Some challenging problems to be worked out