data fusion and multitarget tracking: some interests for military and automobile … · 2013. 8....
TRANSCRIPT
Data fusion and multitarget tracking:
some interests for military and
automotive applications.
Evangeline POLLARD
Just a few words about me…
One year internship
DLR Munich, Germany
Master and PhD degree
Paris/Grenoble,France
Post-doc
Versailles and Sherbrooke (Canada)
Visitor researcher
Berkeley, CA 2
Outlook
1. Military application: convoy detection and tracking
1. Multi-target tracking, a brief overview
2. Hybridization of CPHD filter and MHT
3. Bayesian network for convoy detection
2. Multi-target detection and tracking with uncalibrated
aerial videos
1. Detection
2. Tracking
3. Automotive applications
1. Multi-lane detection and tracking
3
Outlook
1. Military application: convoy detection and tracking
1. Multi-target tracking, a brief overview
2. Hybridization of CPHD filter and MHT
3. Bayesian network for convoy detection
2. Multi-target detection and tracking with uncalibrated
aerial videos
1. Detection
2. Tracking
3. Automotive applications
1. Multi-lane detection and tracking
4
Battlefield surveillance
zone of interest
GMTI data
Videos SAR images
Geographical information System (GIS)
5
General problem
Goal
6
Situation assessment
How many targets on the scene ?
What is their behavior ?
Are they objects of interest ?
Methods:
• 1st step: Using GMTI sensor to detect agregates
• Algorithm weaknesses for closely spaced target tracking
• Use a promising algorithm: the PHD filter (Probability Hypothesis Density)
• 2nd step: Integrate other data types to determine if the detected
aggregates are convoys or not.
Convoy detection
and if so, how many targets are in
Convoy detection
Methods
GMTI data
• GMTI (Ground Moving Target Indicator) data:
• High traffic density
• High maneuverability of ground targets
• Environment complexity (roads, mountains, ...)
• Sensor limitations: measurement noise, spatial and temporal bias,…
• False alarms , PD<1 and spawned targets
7
Single object tracking
solved by Kalman filter equation
with linear/Gaussian assumption
8
Optimal Bayesian Filter: Kalman filter
1
1
11111
1
1)()()(
k
k
kkkkkkk
k
kkkdxZxfxxfZxf
Propagation of the probability density function (pdf) of xk
[Mahler01] : Detecting, tracking and classifying group targets: a unified approach, Proc.
of SPIE Vol. 4380
)(
)()(
)(1
1
1
k
kkk
k
kkkkkkkk
kkk
Zzf
Zxfxzf
Zxf
Prediction
Update
Estimator )(ˆ supargk
kk
x
kkZxfx
a priori pdf
a posteriori pdf
motion model previous pdf
normalization
likelihood a priori pdf
9
Multi-target tracking
State space
• Varying number of targets:
• Birth targets
• Stationary targets
• Output of the observation
zone
• False alarm
• Non-detection
Observation space
solved by MHT,
JPDAF, particle
filter…
and CPHD filter
10
Random Finite Set (RFS)
Target set Xk modeled as a RFS
k
X
kk
X
kkk
kk
BSX
11
)()(11
• Measurement set Zk modeled as a RFS
k
Xx
kk
k
xZ
)(θ
• : Survival targets between iteration k and iteration k-1
• : Spawned targets
• : Birth targets
1kk
S
1kk
B
k
• : Target originated measurement
• : false alarms k
θ
k
11
Multi-sensor/Multi-target Bayes filter
1
1
11111
1
1)()()(
k
k
kkkkkkk
k
kkkdXZXfXXfZXf
Propagation of the joint probability density function (jpdf) of RFS Xk
[Mahler03] : Multitarget Bayes Filtering via First-Order Multitarget Moments, IEEE AES, Vol. 39, No 4
)(
)()(
)(1
1
1
k
kkk
k
kkkkkkkk
kkk
ZZf
ZXfXZf
ZXf
Prediction
Update
Estimator )(ˆ supargk
kk
X
kkZXfX
a priori jpdf
a posteriori jpdf
motion model previous jpdf
normalization
measurement likelihood a priori jpdf
12
13
PHD definition
S
kdxxvSX )(E
• : first-order statistical moment of the multitarget posterior, also
called intensity function or Probability Hypothesis Density k
v
PHD filter principle
(x))()().((x)111 kkkkskk
dvxfPv
kZz
kkdk
kkd
kkdk
dvzPz
vxzP
vPxv
)(.)g()(
(x)).g(.
(x)1)(
1
1
1
• : survival probability between iteration k and iteration k-1
• : transition function knowing the previous state
• : birth intensity
.1kk
f
k
• : detection probability
• : measurement likelihood
• : clutter intensity
sP
dP
)g( xz
k
Prediction
Estimation
14
Several implementation
[Vo06] : Analytical implementation of the Gaussian Mixture Probability Hypothesis Density
Filter, IEEE SP, 2006
15
Le Cardinalized PHD: principle the number of targets is considered as a random variable p
the corresponding pdf is conjointly propagated over time
[Mahler07] : PHD filters of higher order in target number, IEEE AES, 2007
Prediction
Update
Estimator )(ˆ suparg npN
kk
n
kk
a posteriori pdf normalization
Measurement likelihood a priori pdf
Sum of hypotheses for the n targets to be
(n-j) birth l survival (l-j)non survival
16
Labeling
labeling
17
kkN̂
11
ˆ kk
N
Labeled GM-CPHD (1/2)
• : Gaussian set of size
GG
kNiikikikk
Pmw,...,1,,,
,,
Principle
G
kN
kG
• : track set of size describing the target trajectory kk
N̂k
T
kk
NijkikikikksPx
ˆ,...,1,1,,,,,,ˆ
TT
Evaluate the track-to-Gaussian association matrix of size k
A
0
1),( nm
kA
If the Gaussian component n is associated to the track m
otherwise
G
kkkNN ˆ
weight state covariance
state covariance score
Principle
Goal
18
19
• Maximization of the weight matrix 3 tracks
4 Gaussian components
kW
• Minimization of the cost matrix
Contributions to the labeled GMCPHD
kC
If the Gaussian n can be associated to the predicted track m
Otherwise
0
, n k
k
w W
0 1 . 0 7 . 0 0
6 . 0 0 0 0
0 0 0 9 . 0
k W
0 6 . 0 7 . 0 0
1 . 0 6 . 0 7 . 0 0
0 0 0 9 . 0
k W If the Gaussian n can be associated to the predicted track m
Otherwise
0
) , ( n m c k C
0 18 . 5 67 . 3 0
52 . 0 85 . 2 83 . 4 0
0 0 0 55 . 4
k C
Labeled GMCPHD (2/2)
Means
Comparison between the IMM-MHT and the
GMCPHD filter
IMM-MHT GMCPHD
Target position
estimation ++ +
Target velocity
estimation ++ -
Number of targets
estimation - ++
Computational
complexity + ++
Hybridization
++
++
++
+
Creation of a hybrid algorithm which would combine the
advantages of both of them
• IMM-MHT: Interacting Multiple Model – Multiple Hypothesis Tracker
• GMCPHD: Gaussian Mixture Cardinalized Probability Hypothesis Density
20
Hybridization
[Pollard11] : E. Pollard, B. Pannetier, M. Rombaut, “Hybrid algorithms for Multitarget tracking using the
MHT and the GMCPHD”, IEEE Aerospace and Electronic Systems
21
22
Scenario
Convoy velocity: 10m/s
Isolated target velocity: 15m/s
Root Mean Square Error in position
Convoy of 6 targets Independent
target
quality
23
Root Mean Square Error in velocity
Convoy of 6 targets Independent
target
quality
24
Track length ratio
Convoy of 6 targets Independent
target
quality
25
A convoy detection process
Elaboration of a new algorithm combining the advantages of the
GMCPHD and the IMM-MHT with road constraints
+ No performance drop when targets are close together
Aggregate detection
26
27
Convoy: definition and analysis
• Convoy definition
• Number of targets > 2
• Low and constant velocity
• Military type
• Stay on sight
• On the road
• System analyze
• Asynchronous data
• Heterogeneous data
• Random variables
• Missing data
• Temporal evolution
Dynamic Bayesian Network
X1 X2
X3
3
1
321))((),,(
i
iiXPaXPXXXP
Convoy modeling by using Dynamic
Bayesian Network
k-1 k
[Pollard09a] : E. Pollard, B.
Pannetier, M. Rombaut, “Convoy
detection processing by using the
hybrid algorithm (GMCPHD/VS-
IMMC-MHT) and Dynamic
Bayesian Networks ”, Fusion 2009,
Seattle
28
29
Convoy probablity
Time (in s)
30
Number of target estimation
: set of unique value of
Outlook
1. Military application: convoy detection and tracking
1. Multi-target tracking, a brief overview
2. Hybridization of CPHD filter and MHT
3. Bayesian network for convoy detection
2. Multi-target detection and tracking with uncalibrated
aerial videos
1. Detection
2. Tracking
3. Automobile applications
1. Multi-lane detection and tracking
2. Ego-localisation by fusing GPS and proprioceptive data
31
General problem
Detection
Camera motion
Parallax effects with
urban objects
Low image parameters
Unknown camera
parameters
Tracking Extended targets
Probability of detection <1
Hidden zone in urban areas
Spawned targets
High false alarm rate
[Pollard09b]: E. Pollard, A. Plyer, B.
Pannetier, F. Champagnat, G.
Lebesnerais, “GM-PHD Filters for
Multi-Object Tracking in
Uncalibrated Aerial Videos”, Fusion
2009, Seattle
32
Image motion decomposition
Image motion = ground plane motion
+ parallaxe motion
+ moving object motion
residuals
filtered using
size requirements
Optical flow
algorithm
parametric registration
(homography)
33
Ground plane motion (1/3)
Ransac
Region segmentation
Region selection
Dense estimation
mask
Grond plane motion
regions
inliers
Images
Point tracking
Initial transformation
34
Ground plane motion (2/3)
Ransac
Region segmentation
Region selection
Dense estimation
mask
parametric registration
regions
inliers
Images
Point tracking
Initial transformation
[FOLKI]: G. Le Besnerais, F. Champagnat, “Dense
optical flow estimation by iterative local window
registration”, in ICIP’05, IEEE, vol. 1, p. I-137-140
35
Ground plane motion (3/3)
Ransac
Region segmentation
Region selection
Dense estimation
mask
regions
inliers
Images
Point tracking
Initial transformation
parametric registration
36
Ground plane motion (2/3)
Ransac
Region segmentation
Region selection
Dense estimation
mask
regions
inliers
Images
Point tracking
Initial transformation
parametric registration
37
Postprocessing
Image segmentation based
on the norm of the residual motion
After object selection
Selection of moving objects • Edge detection: select region with high density of edges
• Morphological processing: regularize region shape
• Final selection on area (use prior information on object’s size)
38
Detection results
39
GM-CPHD tracking
40
Outlook
1. Military application: convoy detection and tracking
1. Multi-target tracking, a brief overview
2. Hybridization of CPHD filter and MHT
3. Bayesian network for convoy detection
2. Multi-target detection and tracking with uncalibrated
aerial videos
1. Detection
2. Tracking
3. Automotive applications
1. Multi-lane detection and tracking
41
Intelligent vehicle
processing
unit
laser
actuators HMI
camera radar GPS
gyrometer accelerometer
exteroceptif
proprioceptif odometer
42
X
Y
local map
model
• ego-vehicle
• obstacles
• environment - marking lanes
camera
laser
com
scene
Situation assessment
43
Goals
Multi lane
detection
Multi-camera system
X
Y
2
210xaxaay
Number of marking lines
Shape of marking lines
44
Issues
Missing marking line
Identification ambiguity
Curves
Texture changes
Line width change
Shadow
Light condition change
False alarm
45
General scheme 1. Marking feature extraction
Correction
Prediction Association
Propagation
Kalman Filter
Transferable Belief Model
Estimated
marking
Estimated
marking
Classification
Estimated
marking
Road shape
2. Multi-lane detection 3. Estimation
46
Extraction of road markings primitives
Computed only in a region of interest to limit false points
Horizon
Car cover
(height, variance)
Maximum height
[Pollard11a] : Lane Marking Extraction with Combination Strategy and Comparative
Evaluation on Synthetic and Camera Images, ITSC, 2011 47
Local threshold extractor – LT: Local Threshold
– SLT: Symmetrical Local Threshold
Tg
width
LT
SLT
Drawback
• Depends on the width of
the neighborood
• Only horizontal markings
• Sensitive to marks on the
road
48
Background detection extractor – MLT: Median Local Threshold (50th percentile)
– PLT: Percentil Local threshold (43th percentile)
Median filter
Median value 43th percentile value
49
Multi-lane detection
50
Projection according to the road shape
51/52
Multi-lane tracking
Marking lane 2
210xaxaay
state vector
state eq.
Observations
observation eq
solved by Kalman filter
perspective: use of an IMM for dealing with high curve situation [PollardXX]Road Lane Marker Detection and Estimation: a new algorithm and its complete
Evaluation Process, submitted to IEEE ITS
Performance evaluation
53/52
Conclusion
Multitarget Tracking, Situation Assessment,
Object of Interest Detection
Data Fusion, Uncertainty Management,
Evidence Theory, Bayesian Networks
Particle filtering
GMTI, SAR, GPS, GIS, video images
Thank you for your attention!
54
The Joint Directors of Laboratories
[Steinberg98] : Revisions to the JDL data fusion model, Proceedings of SPIE
Multi-target tracking Data fusion
55