In the unlikely event that tho author did not $end UMI a cornpieln manuscript and there are missing pages, these will be noted. A b , if unauthoraed copyright
matefial had to be removed. a witi i n d i i the âeleüm.
Oversize materials (e-g.. maps, drawings, charts) am reproduoed by sediorring
the original. ôegi~ing at the upper left-hand corner and continuing from îeft to
nght in equal secfions with small overlaps,
Photographs induded in Vie minal manuscript have bwn reproduced
xerographically in this wpy. H i g M quality 6* x bladc and white phobgmphic
pnnts are availabîe for any photogmphs or illustmüom a ~ ~ e a r i ~ in Ibis ccipy for
an additional charge. Cultact UMI di- do ofder.
ml& H d l Infomtation and Leaming 300 North Zeeô Road, Ann Arbor, MI 4û1ûSlW6 USA
APPLICATION OF SENSOR DATA FUSION TECHNIQUES TO THE LlGHT ARMOURED VEHICLE
RECONNAISSANCE (LAV RECCE)
Lieutenant Dm van Huyssteen, rmc, BmEngm Canadian Forces
Thesis Submitted To: Department of Electrical and Computer Engineering
Royal Military College of Canada Kingston, Ontario
In Partial Fulfillment of the Requirements For the Degree
Master of Engineering May, 1998
O Copyright 1998 by D. van Huyssteen, Kingston, Ontario This thesis may be used within the Depariment of National Defence
but copyright for publication remains the right of the author.
National Library 1+1 of.,, BiMiitheque nationale du Canada
Acquisitions and Acquisitions et Bibiiographic Services senrices bibfiographiques
395 Weliîngîon Street 395. rue Wdlifghm OitawaON K 1 A W OttawaûN K 1 A M can2da Canada
The author has granted a non- exclusive licence aiiowing the National Library of Canada to reproduce, loan, distribute or seil copies of this thesis in microfom, paper or electronic formats.
The author retains ownershîp of the copyright in this thesis. Neither the thesis nor substantiai extracts fkom it may be printed or otherwise reproduced without the author's pemission.
L'auteur a accordé une licence non exclusive permettaut à la Bibliothèque nationale du Canada de reproduire, prêter, distriiuer ou vendre des copies de cette thèse sous la foxme de mîcrofiche/film, de reproduction sur papier ou sur format électronique.
L'auteur conserve la propriété du droit d'auteur qui protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprimés ou autrement reproduits sans son autorisation.
Abstract
The ability to employ dissimiiar sensors to collect information conceming the behavior of a wide varïety of potential threats is critical to the survivability of high priority military assets. To this end. accurately locating and identifjhg enemy targets foms the basis of an effective response. Consequently, research into surveilIance and tracking systems that best utilize available resources is a high priority.
In this thesis, an efficient, robust and effective target tracking architecture is developed that combines active information h m a Moving Target hdicator (MTI) radar and passive information h m a Forward Looking Infrared @LE) imager. The fmal design, a decentralized architecture, is selected out of a fietd of three candidates on the basis of performance in Monte Carlo simulation. The other candidates are examples of sequential and passive decentratid fùsion- The design is intended as a potential enhancernent to the surveittance suite aboard the Canadian Forces' newly-acquired Light Annored Ve hic le Reconnaissance (LAV Recce).
The design effort inc ludes (i) evaluating various data fbsion architectures; (ii) selecting appropriate tracking filters; (iii) choosing maneuver compensation and track initiation schemes; and, (iv) developing a method to extract a target's aagular position fiom analog infrated images. The scope of this thesis is limited to a simulation of the single target case; however, an extension to the multiple target scenarïo is also discussed in ternis of selecting an appropriate data association algorithm.
No tracking system design is complete without a thomugh field test. To judge the effectiveness of the proposed architecture, the LAV Recce was tested in a realistic surveillance exercise that produced a complete set of pre-processed data with which to validate thïs and f h r e prototypes. In the trial, the radar and FLIR sensors were assigned a c o m o a line-of-sight and both covered 200 milliradian horizontal fields of view, The target, a moving vehicle, was observed h m approximately one kitorneter away. An example of the infrared data is presented, and the image processing techniques adopted in this thesis are demonstrated success fully. The complete test results, which include a test of the proposd tracking architecture with both radai. and FLIR data, will follow in a departmental technical report by the author at a later date.
Acknowledgment
f would Iike to acknowledge the many people who made this thesis possible, especialfy:
1. Mr. .J- Cmickshank and members of the PSSTA Section at DREV for providing documentation, tec hnical and financial support;
. . 11. LCol R. Carruthers and the P M 0 LAV for generous organizational support, and for providing a
test vehicle;
. -. 111. LCoI J. Lord at LFTSC who provided initial documentation and various points-of-contact; and,
iv. Dr. M. Farooq for his constant support, patience and m l v e in guiding the development of this thesis to a successfül conclusion-
Table of Contents
Abstract ..................................................................... i.
. . Acknowledgment ............................................................ -ri-
... Vita ........................................................................ III-
... List of Figures .............................. ,. ............. .. .............. -viii-
List of Abbreviations and P ~ c i p l e Symbols ...................................... -x-
....................................................... Chapter 1 introduction -1- ........................................................ 1.1 Pnmary Objective -1-
........................................... 1.2 Tasks Entailed by Primary Objective 1 . ....................................................... 1.3 Secondary Objectives 1 .
............................................................. 1.4 Assumptions -2- ........................................................ 1.5 Perceived Hurdles -2-
.....*..*.............................. .................... 1.6 Motivation ... -2- ......................................................... 1.7 Related Research -3-
1.8 Background ............................................................. 4 ........................................................ 1.9 Technical Context 4
.................................................... 1.10 Thesis Organization -5-
............................. Chapter 2 The Tracking Problem and the Kalman Filter -6- .......................................... 2.1 Discrete Linear Kaiman Filter (DKF) -6-
.............................................. 2.2 Properties of the Kalrnan Filter -10- ........................ 2.2.1 Covariance Properties and Monte Car10 Simulation -10-
................................................ 2.2.2 Innovation Properties -11- .............................................. 2.3 Extended Kalrnan Filter (EKF) -11-
..................................... 2.4 Functional Elements of a Tracking System -13- .................................. 2.4.1 Maneuver Detection And Compensation 13-
2.4.2 TrackInitiation .................................................... -14- ................................................... 2.4.3 Data Association -14-
2.4.3.1 Measuement Oriented Techniques ............................... -1 S- ..................................... 2.4.3.2 Track Oriented Techniques -16-
................................ 2.4.4 Architectures for Kinematic Data Fusion -17- .............................. 2.4.4.1 Decentralized (Sensor level) Fusion -17-
.......................... 2 .4.4.2 Centralized (Measurement Level) Fusion -18- ............................................ 2.4.4.3 Sequential Fusion -19-
............................................. 2.5 Short List of Design Approaches -19-
Chapter 3 Detailed Evaluation of Candidate Architectures .......................... -21 . 3.1 Candidate SF-1 : Sequential Fusion ........................................ -21 .
3.1.1 State and Measurement Models ........................................ -21- 3.1.2 Filter Formulation .................................................. -23- 3.1.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-
3.2 Decentralized Architectures ..............................................~.. -25- 3.2.1 Candidate DF-1: Static Decentralized Fusion ............................. -26-
3.2.1.1 Background ............................................... ..2 6- 3.2.1.2 Application of Static Fusion to the A L I S S ........................ -27- 3 .2.1 -3 Generating Local Pseudoestimates ............................... -28- 3.2.1.4 Rem& ................................................... -30-
. ............................ 3 .2.2 Candidate DF-2: Static Fusion with Feedback -31 ..................... 3.2.3 Candidate DF-3: FuIly Decentralized Dynamic Fusion -32-
3.2.3.1 Timing Considerations ........................................ -33- ................................... 3.2.4 Remarks on Decentralized Approach -35-
......................... 3.3 Replacing Standard EKF with More Efficient Formulation -35- ......................................... 3.4 Data Association and Track Initiation -35-
3.5 Data Association ......................................................... -35- .................................... 3.5.1 Nearest Neighbour Data Association -37-
................. 3.5.2 Application of NN Data Association to Decentralized Fusion -39- ................... 3 S.3 Application of NN Data Association to Sequential Fusion -40-
3.6 Track Initiation ........................................................... -40-
Chapter 4 i n h e d Image Processing .......................................... 42- 4.1 Mapping Between Pixel and Angle Coordiaate Systems ........................... -42- 4.2 Extracting the Target Ceatroid from a Noisy Image ............................... -44-
........................................ 4.3 Interpreting the Analogue Video Signa1 -45- ................. 4.4 Single Frame Estimation: Determining Centroid Mean and Variance -47-
...................................................... 4.4.1 Background - 4 7 - 4.4.2 Application of a Bandpass Filter ...................................... - 4 8 - 4.4.3 Clustering ........................................................ -51-
Chapter 5 Simulation of Design Alternatives .................................... -52- 5.1 Scenario ................................................................- 52- 5.2 Candidate SF- 1 : Sequential Fusion (Case 1 and Case 2) ........................... -54- 5 -3 Candidate DF- 1 : Static Decentralized Fusion ................................... -56- 5.4 Candidate DF-3: Dynamic Decentralized Fusion ................................ -58- 5.5 SelectionofArchitecture ................................................... -61-
Chapter 6 Validation with Real Data ........................................... -62- 6.1 Measures of Performance ................................................... -62- 6.2 Setup and Procedure ....................................................... -63- 6.3 Infrared Data Processing ................................................... -63- 6.4 RadarData ............................................................... 68-
Chapter7 Sumrnary .................................................... -70- 7.1 Objectives Met ........................................................... -70-
................................................. 7.2 Results and Design Choices -70- ................................................ 7.3 Novel Features of this Thesis -71-
................................... 7.4 Recommendation for Continued Investigation -72-
Annex A Project Background ............................................ -73-
Annex B An Altemate Technique for Transfemng Angular .......................... Measurements Between Coordinate Frames -74-
................................ References ,,., . -78-
List of Tables
Table 2-1 The Discrete Linear Kalman Filter (Dm ................................ -8-
Table 2-2 Extended Kahan Filter (EKF) ........................................ -12-
Table 2-3 Short List of Design Appmaches ....................................... -20-
............................................... Table 3-1 Modified-Baheti Filter -36-
Table 3-2 Validation Matrix for Hypothetical Trackiag Scenario ...................... -38-
..................... TabIe 3-3 Assignment Matrix for Hypothetical Trackhg Scenario -39-
List of Figures
Figure 1-1
Figure 2-1
Figure 2-2
Figure 2-3
Figure 3-1
Figure 3-2
Figure 3-3
Figure 3-4
Figure 3-5
Figure 3-6
Figure 3-7
Figure 3-8
Figure 4- 1
Figure 4-2
Figure 4-3
Figure 4-4
Figure 5-1
Figure 5-2
Figure 5-3
Figure 5-4
Figure 5-5
Figure 5-6
Figure 5-7
Figure 5-8
Figure 5-9
Figure 6-1
Figure 6-2
Figure 6-3
Figure 6-4
Figure 6-5
Light Armoured Vehicle (Reconnaissance) Version ........................ -3-
Discrete Linear Kalman Filter ......................................... -9-
DecentraIized (Sensor Level) Fusion .................................... -18-
Centraiized (Meamrement Level) Fusion ................................. 19-
Relationship Betweea State and Measurement Coordinate Systems ............ -22-
Sequential Architecture for A\LRSS .................................... -25-
Image P b Coordinate Systern ....................................... -29-
Mismatch Between Linear Cartesian and image Plane Dynamics .............. -29-
Static Fusion with Feedback .......................................... -32-
Dynamic Decentralized Fusion ........................................ -33-
Time Alignment at Local And Global Levels ............................. -34-
Sample Nearest Neighbour Data Association Scenario ...................... -38-
Target Location in the Image Frame .................................... -43-
Relationship Between Local and Globai Coordinate Frames ................. -44-
Application of a Bandpass Filter ....................................... -48-
Application of a Bandpass Intensity Filter Given Non-Zero Background ........ -50-
Simulated Course Profile for (a) Imager and (b) Radar ...................... -53-
Sequential Fitter Performance (Total Position) ............................ -54-
Sequential FiIter Performance (Y and X Position) .......................... -55-
Sequential Filter Performance (Total Position) ............................ -56-
Static Decentralized Filter Performance ( i and j image Position) .............. -57-
Static Decentralized Filter Performance (Total Position) ..................... -58-
Dynamic Decentralized Filter Performance (Total Position) .................. -59-
Dynamic Decentralized Filter Performance (Total Position) ................. -60-
Dynamic Filter with Range Propagation (Total Position) .................... -61 . .................................................... TargetScenario a
Infrared Video Image Captured by LAV Recce ............................ -64-
Histograrn of Typical Pixel Intensities .................................. -65-
Intensity Bandpass with Overly Wide Threshold .......................... -66-
Intensity Bandpass for PT > 0.95 and P, < 0.05 ............................ -66-
Figure 6-6 Clustered Pixels .................................................... -67- Figure 6-7 Isolated Centroid ................................................... -68-
Figure 6-8 Radar Setup ............-.......................................... -69-
List of Abbreviations and Principle Symbols
Abbreviations
AlLRSS
EE
FOV
IFOV
FLIR
JPDA
LAV Recce
LRSS
ME
MOP
MSE
MSTAR
MTI
NEES
NIS
NN
PDA
RMSE
Advanced LAV Recce Surveillance Suite
Expected Error
Field of View
Instantaneous Field of View
Forward Looking Infrared
Joint Probabilistic Data Association
Ligbt Armored Vehicle Reconnaissance
LAV Recce Surveillance Suite
Measurement Error
Measue of Performance
Mean Square E m r
Man-Portable Surveillance and Target Acquisition Radar System
Moving Target Indicator
Norxnalized Expected E m r Squared
Nonnalized Innovation Squared
Nearest Neighbour
Probabilistic Data Association
Root Mean Square Error
Principle Symbols (And Chapter Where First Used)
Estimated quantity (Chapter 2)
E m r tenn (Chapter 2)
Confidence interval (Chapter 2)
Background extent (Chapter 4)
Binary value of pixel (i j) (Chapter 4)
Pseudoestimate (C hapter 3)
Nonnalized Innovation Squared (MS) (Chapter 2)
Inadiance (Unit flux incident per unit area) (Chapter 4)
Observation matruc (Chapter 2)
Gcayscale value of pixel at coordinate (i j) (Chapter 4)
Vertical pixel coordinate (Chapter 4)
Horizontal pixel coordinate (Chapter 4)
Discrete time index for tirne t, (Chapter 2)
Micates at time t, given information up to and including t, (Cbapter 2)
Status up to and including time km, (Chapter 2)
Radiance (Chapter 4)
Probability of detecthg a target pixel (Chapter 4)
Probability of detecting a background pixel (Chapter 4)
Probability of detection (Chapter 4)
Probability of false alarm (Chapter 4)
Kalrnan Filter covariance matrix (C hapter 2)
Elevation (Chapter 4)
Enor fiinction (Chapter 4)
State noise covariance matrix (Chapter 2)
Measurement noise covariance matrix (Chapter 2)
innovation covariance (Chapter 2)
Variance (Chapter 2)
Continuous time index (Chapter 2)
Transpose operator (C hapter 2)
Target extent (C hapter 4)
Bearing (C hapter 4)
Time difference (C hapter 2)
State noise vector (Chapter 2)
Measurement noise vector (Chapter 2)
State vector (Chapter 2)
Chi square test statistic with n, degrees of fkedom (Chapter 2)
Position in 'x* coordinate (Chapter 2)
Position in 'y' coordinate (Chapter 2)
Position in 'z* coordinate (Chapter 2)
Observation vector (Chapter 2)
Chapter 1 Introduction
The ability to employ dissimiiar sensors to collect information concerning the behavior of a wide variety of
potential threats is critical to the sunrivability of high priority military assets. To this end, accurately
iocating and identi-g enemy targets forms the basis of an effective response- Consequently, research
into surveillance and tracking systems that best utilize available resources is a high priority.
1.1 Primary Objective
The objective of rhis thesis & fo d d o p an eflcient, r o k t and Meciive target tracking
architecture that combines active (range, bearing and elevation) information fiom a Moving Target
Indicator (MTI) radar and passive (bearing and elevation) information fiom a Fonvard Looking
Infrared (FLIR) imager.
The design of the architecture is intended as a potential enhancement to the surveillance suite aboard the
Canadian Forces' newly-acquired Light Armored Vehicle (Reconnaissance version). The "LAV Recce"
is the anny's primary surveillance asset at battk and brigade group levels. Its mandate is to provide
information on the enemy under al1 environmental conditions, across the fidl range of conflict, h m low
intensity to high intensity operations.
1 9 Tasks Entailed by Primsry Objective
Satisvng the prirnary objective entails:
1. Evaluating appropriate h i o n architectures and cboosing one on the basis of robustness.
computational efficiency and effectiveness; . . II. Selecting an appropriate tracking filter (or filters, depending on the architecture);
iii . Choosing maneuver detectiod compensation and tmck initiation schemes;
iv. Deveioping a method to extract a target's angular position h m analog infrared video images;
v. Evaluating the performance of the candidate algorithm via Monte Car10 simulation.
1.3 Secondary Objectives
This thesis constitutes the initial stage of a more comprehensive pmject commissioned by the Defence
Research Establishment Valcartier (DREV). The thesis is limited to a simulation of the single target
scenario while the larger project will incorporate an extension to the multiple target environment (by
adding a data association module), and include a validation of the algorithms with actual surveillance data.
Initial progess toward meeting the secondary objectives is documented in this thesis.
1.4 Assumptions
The architectures presented in this thesis assume the availability of the LAV Recce's Ku band doppler
radar (Surveillance mode) and 8-14 micrometer FLIR (wide FOV setting). For simplicity, the sensors are
assumed to be bore-sighted and stationary- The target profile(s) are constrained to travel at constant
velocity. To avoid unnecessary encumbrances, track initiation is assumed to have k e n completed at the
radar level. For the sake of generality, filter measurement models are developed in spherical coordinates,
even though the LAV Recce's MSTAR doppler radar provides only range and bearing information. The
limiting polar case for al1 equation can be obtained by setting the elevation component to zero. The host
platform for the proposed system Ml1 be referred to as the 'Advanced' LAV Recce Surveillaace Suite
(A\LRSS),
1.5 Perceived Hurdles
The LAV Recce is an operational vehicle, and as such modifications to the existing setup are not trivial.
A particular concem is the fact the sensors' displays are analog. The pmposed system, on the other
hand, requires a digital representation of the plan position indicalor (PPI) display, and digital FLIR image
sequences. Thus. the sensor interfaces would have to be modified to accommodate the proposed tracking
system. However, from a feasibility point-of-view, a radar demoistrator (based on the MSTAR) bas
already been configured at DREV that pmvides digital data- The FLiR images can be converted with a
' framegrab be r' .
1.6 Motivation
The LAV Recce (Figure 1-1) presently has no target tracking facility. While the addition of a simple
radar tracker represents an irnprovement in itself, this thesis demonstrates how an infiared imaging sensor
may be incorporated into the tracking loop to increase the precision of the estimation process. The
improvement relates to the fact that optical devices typically provide much better angular resolution than
radar. Furthexmore, themion of multipk sensors offers potentialiy more robust operational performance
and increased spatial/ temporal coverage compared to single seasor systems. Finally, automating the
overall tracking process will d u c e the operator's workload and d u c e fatigue-relatai errors.
Figure 1-1 Light Armowed Vehicle (Reconnaissance) Version
1.7 Related Research
A group at the Defence Research Establishment Valcartier (DREV) is currently investigating 10 ptential
improvements to the LAV Recce's Surveillance Suite (LRSS). While the proposed tracking algorithm is
envisaged as part of the 2003 mid-life upgrade, the feasibility of the pmject (in tenns of required
hardware/ sensor moditications) bas already been identified by one of the group members Cl]. While this
work was initiated by the author at the Royal Military College, it has since received both financial and
technical support iiom DREV. As mentioned, this thesis constitutes the initial stage of a more
comprehensive project that will be delivered to DREV as a technical report-
1.8 Background
The enhancement of a multi-sensor surveillance platfonn must be preceded by a detailed analysis of the
candidate system, and incIude a description of both its cunent roies and potentia1 capabilities. Initiai
documentation for this project was obtained k m the Land Forces Technical Staff College (LFTSC), the
Project Management Office (PM0 LAV) at National Defence Headquarters (NDHQ), and h m
Defence Research Establishment Valcartier [SI, 131, [4]. After gaining an initial understanding of the
current system's strengths and weaknesses, a meeting was held with two technical authorities h m
DREV to discuss adding a mdimentary target trac king facility b a s 4 on both the radar and infî-ared
sensors. The consensus is that the LAV Recce Surveillance Suite (LRSS) incorporates a set of vety
capable sensors (including a FUR, day carnera, doppler radar, laser range finder and audio input
interface). but is Iimited in its ability to exploit them to hl1 advantage. After identifjhg the general area
for investigation, fiirther inquiry was conducted to ground the development in a practical environment, and
to collect typical surveillance data with which to test the prototype system. A chronology of the
background preparation is attached as Annex A.
1.9 Technical Context
The design of the proposed system incorporates several mathematical methods h m the emerging field of
data fusion. In the present context, data fusion refers to a 'multilevel, multifaceted process dealing with
the detection, association, correlation, estimation and combination of data h m multiple sources to achieve
(i) refined state and identity estimates and (ii) complete and timely assessments of situation and threat'
[5]. A mode1 has been devised by the Unites States' Joint Directorate of Laboratories (JDL) to
categonze potential systems according to this definition (Table 1-1). As the goal of this thesis is to track
targets by fusing kinematic information (range, bearing, elevation) h m active and passive sensors, it
relates to Level 1 Fusion. Two other levels have also been designated [6]; however, they concem more
abstract issues (situational assessrnent and threat analysis) that do not apply to this thesis.
Level 1 - "Inference Level"
1 Objectives:
Functions - Level 1 -
Data alignment
Techniques - Level 1 - l
Coordinate transfonns Augmented rotation mapping l
Correlation Gating techniques Multiple hypothesis tracking (MHT) Joint pmbabilistic data association
(RDA) Nearest neigtibour 0
Kinematic attribute estimation
Recursive estimation (filters) Kaiman, a-& Interactive tIMM)
Batch estimation Maximum likelihood Hybrid melhods
Object identity estimation
Physical models Feature-base techniques
Neural networks Cluster algorithms Pattern recognition
S yutactic models Table 1-1 Description of Level 1 Data Fusion [6]
1.1 0 Thesis Organization
The thesis is organized as follows: Chapter 2 discusses the tracking problem. in particular the Kalman
filter is detailed as it foms the backbone of most modem real-time filtering systems, and because it is
used extensively in data fusion formulations. Chapter 2 also outlines the notion of data association and
introduces standard multisensor filtering architectures. In Chapter 3, a set of three application-specific
architectures are developed as candidates for the AILRSS. These architectures assume the FLiR
provides angular measurements of the target location. A method to generate these angles from FLiR
imagery is presented in Chapter 4. In Chapter 5, the candidate architectures are tested via Monte
Carlo simulation. The purpose of the simulation is to determine which architecture is most suitable to the
AILRSS. Chapter 6 describes a live trial perfomed to record actual data for validating the adopted
prototype (chosen in Chapter 5). The thesis concludes in Chapter 7 with a summary and
recornmendations for fûrther investigation.
Chapter 2
The Tracking Problem and the Kalman Filter
This thesis deals with many aspects of the tracking problem, h m target detection to data association.
Although they differ in fùnction, most of these areas rely in some way on the pmperties of an invaluable
mathematical tool - the unbiased linear minimum mean square e m r (MMSE) estirnator. This chapter
begins by introducing the most prolific MMSE algorithm - the Kalman filter- considered the workhorse
of tracking [7]. The statisticai properties of its output are discussed next, followed by a sketch of non-
linear filtenng in tenns of the extended Kalman filter. The remainder of the chapter concentrates on the
supporting elements of a cornprehensive target t r a c h g system, De facto techniques for maneuver
detection and compensation are outlined, as are standard approaches to architecture and data
association. These issues are revisited in Chapter 3, which expands on the key issues to motivate
appropriate design choices-
2.1 Discrete Linear Kalman Filter (DKF)
Generating an 'optimal' estimate of a system's state, X(k), from a set of noisy observations is a
classical filtering problem, initially champioaed by Gauss in his shidy of celestial orbits [8]. In general, a
state is an n-vector whose elements are determined completely by their previous instances; for
example, the state of a tank negotiating a course on the battlefield can be described in Cartesian
coordinates by
X ( k ) =[m ~ ( 4 dk) ~ ( k ) ml mlr (2- 1
where x(k), y(k) and z(k) represent the tank's position at time tk; *(k), j(k), and i ( k ) its velocity,
and where T denotes the matrix transpose operation. The evolution of a discrete system with linear
dynamics is described by the difference equation:
x(k + 1) = ~ ( k ) x ( k ) + ~ ( k ) ; ~ ( k ) - N ( o , M ~ ) ) k = O,I, ... (2-2)
where X(k+l) is the state at time t,,,; F(k) the n x n transition matrix; and where w(k) is a zero mean
Gaussian noise sequence reptesenting a random disturbance of variance Q(k)zO. in the previous
instance, if the target travels with constant velocity then ~ ( k ) = for AT 5 b, - t, and I3
= 3 x 3 identity rnatrix. Typically, not al1 elements of the system state are observable. Assuming that
the measurements are made in the presence of zero-mean Gaussian noise, then the m x 1 measurement
vector Z(k) is defined by:
~ ( k ) = H(L)X(L) + v(k); v(k) - N(O. ~ ( k ) ) (2-3
where H(k) is the n x m observation m a e and where ik) is the noise sôquence (with variance R(k)
2 O). For example, if it is only possible to record position in X-Y-Z coordinates, then H ( k ) = [I , 41.
The goal of the tracking problem is to estimate the desired state (position in this case) based on the
measurement process and knowledge of the target's dynamics. The 'optimal' solution minimizes the
mean square error (MSE) between the true state X ( k ) and the estimated state k ( k l k ) .
Specifically, it is required that
E [ ( x ( ~ ) - R(k~k)&' (k ) (x (k ) - k(k lk ) ) r ] = min
where S(k) is an arbitrary n x n weighthg matrix. Although Wiener formulated a MMSE algorithm in
the fkequency domain for Iinear the-invariant systems, it was Kalman who finally refomulated Gauss's
earlier resul ts to produce the ticst time domain recursive estimator for linear time varying (LTV)
systems. The equations describing the so-called Kalman filter can be developed according to
Bayesian, maximum Iikelihood or least squares philosophies. These derivations are well documented
[9],[10] and al1 result in an identical algorithm, which is summarized in Table 2-1 [9], accompanied by a
list of pertinent definitions and notation.
System mode1
Measurement mode1
Initialization
Prediction (propagation) equations
Correction (update) equations
Assumptions
Mscrete Linear Kaiman Fitter
k(w) = initial estimate o f state vector P(w) = initial estimate of e m r covariance matrix
k(k lk - 1). P(kk-1) n estimate
k time index (present a priori state vcctor and estimate
instant) error covariance matrix F(k) state transition matrix
H(k) observation rnatrix m k ) , P(kF) K(k) Kalman gain matrix
a posteriori state veetor and estimate (k) vector crror covanance matrix w(k) process noise vcctor
v (k) measurcment noise vector
Table 2-1 The Discrete Linear Kalman Filter (DKF) [9]
While an exhaustive derivation of the Kalman filter equations is outside the scope of this work; a
description of its mechanics is in order. At the h a r t of the filtering process is the construction of the
-8-
gain matrix, K(k). The gain is inversely proportional to the expeaed error of the estimate, P(k[k),
which is an n x n mavix whose ij" element is given by
P,(kl k) = E {(xi(k) - ~ ' ( k l k ) X ~ ' ( k ) - k i ( k l k ) ) ) (2- 15)
where i and j represent one of the n elements of the state vector. For instance, the expected e m r (EE)
in the x direction of the sptem described by Equation 2-5 cornponds to element i = 1, j = 1 of the
covariance matrix, while the expected em>r in the y direction corresponds to P&). Given the
expected error of the propagated (predicted) state estirnate, the pin can then be calculated. The gain
is then used to form the a posteriori state estimate as the optimal combination of the a priori state and
the present measurement (Equation 2-10). The a priori estimate X(k1k-1) is simply a propagation of
the previous update according to the assumed dynamics, and is thus the best estirnate at time k given
measurements until tïme k- 1 -
Figure 2-1 Discrete Linear Kaiman Filter
X(k+l lk) SYSTEM
X(klK)
9-7 z(k) A
> PREDICT FILTER . > P(klk)
. P(k+l lk)
w(k) v(k) A
(2-1 2) (2-13) i ., CORRESPONDING KALMAN FlLTER EQUATIONS ( )
Because it is recursive, the Kalman filter may be mechanized as a ceai-tirne algorithm (Figure 2-1).
Initial values for the state vector and covariance matnx must be assumed to begin the process. While
the most accurate information available should be used to initialize the state, the covariance is usually set
to a conservatively high value to insure gain does not increase t w quickly. High uncertainty resdts in a
Iow gain, which weights the measurements more heavily than the estimates. As the filtering process
continues the covariance decays exponentially toward zero, and the estimate receives increasingly more
weight. Thus an accurate system mode1 is critical. In practice, a 'track maintenance' facility must be
included to manually break the filter loop in the case where the target dynamics are no longer consistent
and the filter has diverged Loosely speakuig, divergence is said to have occurred when the difference
between true and estimated states increases over time.
2.2 Properties of the Kalmrn Filter
2.2.1 Covariance Propefties and Monte Carlo Simuîation
The fact that the Kalman filter calculates the expected estimate error (EE) is a very important property.
Specificatly, it allows the filter's effectiveness to be detennined by conducting a set of independent runs
and calculating the mean square e m r (MSE) between true and estimated states for each sample time.
Specifically. if r = 1 .. R nias are conducted, then the MSE for the estimate of the i element of the state
vector is calculated by
As R tends toward infinity, the MSE will converge on the expected error, P(klk), provided the system's
dynamics have been properly modeled; Le., for the same element of the state vector,
ei(klk) = E { ( x ' ( ~ ) - i i ( k l k ) ) ' }
This technique, called Monte CarIo simulation, is used extensively in this thesis to test the effectiveness of
candidate filters and architectures, Note that the best possible estimate of a linear system by any filter
corresponds to the Cramer Rao Lower Bound, which is equivalent to the expected error, P(wk) [ 1 1 1.
2.2.2 Innovation Properties
The difference between the measwed state and the estimated States over time forms the Gaussian
disûibuted innovation process, defmed by Z(k) , where
Nomalizing the innovation pmess with respect to its variance, S(k), results in a chi-square distributed
quantity with n= degrees of fkedom, where n= is the dimension of the measurement vector. This property
is important since it provides a standard basis for cornparhg the 'proximity' of the estimate to the true
state, which is usehl in detecting maneuvers- Specifkally, the normalized innovation squared (NIS)
sequence (denoted by q,j may be calculated by dividing the square of the latest innovation by its variance;
i.e,
~ ( k ) = ZT(k)S'(k)Z(k) k = 12.3, ... (2- 19)
where
The NIS accounts for the uncertainty in both the present rneasurement and the latest predication. For a
correctly modeled system, the probability that the NIS falls within tolerance X2, is given by
where (1 -a) is the desired confidence, and X\ is the chi-square test statistic for the given dimension
(available from standard mathematical tables 1121). This test also forms the basis for the gating pmcess in
data association, which is freated briefly in tliis thesis in Section 3.5.
2.3 Extended Kalman Filter (EKF)
In Section 2.1 the DKF was fonnulated as the optimum linear estimator. Frequently, however, the state
and measurement equations are non-linear, and are descrikd by the more general equations
X(k + 1) = f ( ~ ( k ) , k)+ w(k) ; w(k) - ~ ( 0 , QW) (2-22)
and
~ ( k ) = h ( ~ ( k ) , k ) + v(k); v(k) - N(O, ~(k)) (2-23)
where f ( ~ ( k ) , k ) and b ( ~ ( k ) . k ) are non-linear functions of the state. To obtain a tractable
solution to the estimation problem, it is essential tbat the non iinearities be expanded by linear
approximation. 'The most common approach is to employ a power senes mund k(&lk - 1). i.e.,
This yields the Extended Kalman Filter (EKF), whose equations are presented in Table 2-2. The EKF
provide a recursive means of computing an approximation to the MMSE estimator of X(k), despite non-
linearities in the system andl or state mode1 [9].
Extended Kalman Filter (EKF)
System Measurements
k(w) = initial estirnate of state vector
P(O(O) = initial estimatc o f error covariance rnatrix
Predic tion
Correction
Defini tions
Table 2-2 Extended Kalman Filter (EKF) [9]
Although the EKF is similar to the linear DKF, it also differs in two cntical ways: First, its precision is
-12-
governed by the number of higher derivative temis kept in f(x(k1.k) and L(x(&). k ) . Second,
its gains cannot be pre cornputeci. As might be imagined, the EKF can be computationally expensive.
Possible remedies to these two independent problems include (i) reformulating the filter as a nonlinear
estimator in the case of large nonlinearities, or more simply, retaining higher derivative ternis in the
power series expansion; and (ii) expanding the measurement and state Jacobians (Eqetions 2-35,36 )
around a pre-set pseudolinear trajectory (which will allow the gain to be precalculated and stored at the
cost of increased bias).
in the present application, computation requirements are helped by the presence of a linear state model;
however, a further reduction in computation can be achieved by employing a slightly suboptimal
reformulation in the Modifieci-Baheti filter (Chapter 3) [13].
2.4 Functional Elements of a Tracking System
Up to this point, Chapter 2 has concentrateci on defining the optimal estimator for the target state under
the assumption that the target has been identified and that it's dynamics are completely specified. in
practice, this cannot be guaranteed. A reaiistic course profile will be time varyhg; thus it becomes
necessary to include a mechanism to not only detect the onset of a maneuver, but also to compensate
for it. A fully functional and robust tracking system must also include several components apart fiom the
filter itself A track initiation scheme is necessary to begin the estimation process and to generate the
initial state, while a data association facility is needed to assign measurements to their origin when
tracking in clutter and to resolve the multiple target case. Finally, the non-trivial issue of choosing an
architecture must be addressed before the design can kgin in eaniest. These issues are now discussed
with the goal of elucidating appropriate strategies for the A\LRSS.
2.4.1 Maneuver Detection And Compensation
In principle, there are many ways to accommodate a maneuvering target profile. Generally, these
techniques may be divided Uito two categones: those that assume a priori knowledge of the maneuver,
and those that do not. in the first case, the interactive Multiple Mode1 (IMM) strategy [14] employs
several filters to follow a target, each according to diffemt set of assumed wget dynamics. Bar
Shalom has akso proposed a variable dimensional filter that switches between constant course and
constant acceleration state models at the onset of a maneuver [15]. To some degree, both of these
techniques require knowledge of the target's bue motion.
However if no a priori kmwledge is available, it is still possible to handle maneuvers by includhg a
noise term in the state mode1 (see Equations 2-5,2-12). This effectively prevents the filter gain fkom
decaying to zero, which would result measurement process receivùig inadequate weight. Altemately, if a
maneuver is detected (by m o n i t o ~ g the innovation process), the filter's covariance matrix, P(ktk), can
be reinitialized to favour the measurements and reacquire the track. Because of their simplicity, the
latter two techniques will be considered the primaty candidates for application to the AILRSS. In the
case of a decentralized architecture (discussed later), separate filters are used to estimate the state at the
local (sensor) level and at the global level. In this case, maneuver detection and compensation would be
carried out at the sensor level; Le., the iocal filters' covariance matrices would be reinitialized.
2.4.2 Track Initiation
Before the kinematic properties of a target can be esthatecl, it is necessary to acquire the track in the
first place. Track initiation can be accomplished as a side effect of data association algorithms which
implicitly assume that each measurement might originate tiom a new target. Otherwise, a separate
mechanism must be incorporated to match the development of a profile to the assumed target dynamics.
Essentially, the process involves predicting the next location of the target based on the previous
measurements. If the predicted and rneasured points coincide, then the track is initiated.
2.4.3 Data Association
When operating in a cluttered andl or multitarget environment, the ability to assign measurements to their
sources is essential. Various techniques exist to accomplisb the socalleci data association; generally
they fa11 within one of two categories: measurement oriented techniques and track oriented techniques.
While the suitabiiity of one method over the other is case dependent, it is generally true that
measurement oriented techniques offer greater pediommce at the pnce of higher computational
requirements and complexity, while track oriented techniques still provide a potentiaiiy robust meam of
attributhg measurements to target tracks yet with significantly less overhead. The latter approach is
more attractive for the present design. A briefdescription of the candidate methodologies helps put hto
context the eventual choice, as detailed in Chapter 3.
2.4.3.1 Measurement 0- Techniques
The predominant feature of measurement oriented data association is iîs ability to defer the ultimate
assignment decision uotil a desired confidence level is met. Practical algorithm are usually a variant of
the so-called Multiple Hypothesis Tracker (MHT) 1161, which makes strictly one-to-one measurementl
target assignment under the assumption that they are exact- Specifically, each new measurement zi E 2,
collected at time k is assigned a set of hypotheses, R(k), such that
T( j ) 3 meas belongs to existing track i = l NT mcas bclongs to new track i = 2
FA s meas is false alarm i = 3
where Tu) represents the jrh track. in hi@ clutterl high target count environments, the MHT becomes
unwieldy over time because new tracks are required for every measurement to account for each
hypothesis. To reduce the computational bwden, various methods have been proposed to elirninate or
'prune' established track files that are probably not correct. The probability of association of each
hypothesis Q(k) given the present measurement set Z(k) E (z,, z2, ... ,zk} is a factor of the probability
of detection, PD, the density of false alarm, BFA , and the density of new targets, 8, [L6],[17].
Assuming the sensor reports the number and location of each target, the probability of association is:
P ( k ) = P ( . ; ( W ( k ) ) (2-3 7)
Apart fiom its potential accuraçy, the MHT offers the obvious advantage of initiating and maintaining
tracks implicitly in the association pmcess; this is not the case with any of the track oriented appmaches.
In practice, the MHT has k e n applied most extensively to highly dedicated tracking systems which
employ a measurement level fiision topology (centralized or sequential).
2m4m3.2 Tnck &iented Techniques
Track oriented techniques emphasize computatiod ef'fïciency over exhaustiveness. Whereas the MHT
considers al1 previous measurements in the assignwnt process (and actually defers making the decision
until enough additional information has been collecteci to guarantee the correct decision), track oriented
techniques use only the most recent observations to make irrevocable assignments for a set number of
targets. Attendant methods to begin the tracking process and maintain the tracks are thus necessary if a
track oriented approach is used; these are described in Chapter 3. Several variations of track oriented
data association procedure exist, ranging h m the mdimentary nearest neighbour (NN) algorïthms to the
significantly more complicated Joint Probability Data Association (JPDA) f r l t e ~ A brïef description of
each is given here.
Nearest Neighbour (NN) Data Association [161. As the name suggests, the nearest neighbour
approach makes associations according to a rule that minimizes the collective distance between al1
measurements and tracks. Only measwements within a precalculated gate are considered for association,
and in the absence of a candidate observation, the track may be either dropped or updated with the
predicted state. The nearest neighbour can assiga each measurement to at most one target, and practical
implernentations require the that the number of associations is maximized Employing this rule leads to
conservatively suboptirnal results; however, the algorithm becomes much more efficient as it reduces
number of potential observation-to-track scenarios. The main weakness of the temporarily and spatialiy
'hard' sequential NN is that it can fom its decisions according to a marginal confidence level, and c m not
accommodate situations requiring that one measurement be shared by multiple targets. Thus, the NN is
not well suited to situations with several closely spaced (and not consistently resoIved) targets. To
alleviate some of its weaknesses, various track-splitting methods have been proposed. These
modifications create altemate tracks which improve the chance of correct assumptions in ambiguous
conditions.
Probabilistic Data Association (PDA). The PDA algorithm improves upon the nearest neighbour
approach in that it is not limited to assign at most one measurement to each target. That is, every
measurement within a track's validation gate will be used for the next update, each weighted by the
reciprocal of their respective distances to the gate centre (i.e, the location of the last a priori estimate.
Thus, the PDA performs averaging over every observation-to-track hypothesis that has roughly the same
likelihood 1181. There is, however, no guarantee that the PDA will yield better results than NN, especially
in high signal-to-noise/ low clutter Environments.
Joint Probabilistic Data Association (JPDA). The PDA or 'all neighbour' strategy is noteworthy as it
allows tracks to share measurements. The JPDA improves upon the PDA in the rnultitarget sceuario by
also letting measurements share targets. Its key is the evaluation of the conditional probabilities of these
two joint association events; specifically, a variable, 0, is used to account for al1 possible target/
measurement associations [ 161, where
for
measurement j on'ginates 9, = P J = I .... mk t = O ..... T
fiPm target r
Assuming that validation gates are not used to eliminate unlikely measurements h m consideration. then
the JPDA is equivalent to a zero scan back MHT, minus track initiation- As such. the joint events are
akin to the MHT hypotheses. The JPDA was developed at the same time as the MHT, and like its rival,
has been applied in various forms to specific problems [19]. While it is versatile the JPDA can also be
quite cumbersome, and for this reason sometimes loses its appeal.
2.4.4 Architectures for Kinematic Data Fusion
The goal of a typical tracking system is to calculate the minimum variance estimate of a target's state.
By adding redundant sensors to a surveillance platforni, it is possible to reduce Mher the estimation
error. However, the mathematical solution to the MMSE problem in the case of multiple sensors may be
achieved by more than one mathematically equivalent formulation, which is important because an efficient,
robust and effective overall system depends on a judicious choice of architecture. To this end, identifiing
a candidate architecture at an early stage of the design is criticai, as the choice bas an impact on the
attendant selection of data association and track initiation mechanism. In the next section, decentralized,
centralized, and sequential architectures are introduced, as they are discussed in 1161, [20], and [21].
2.4.4.1 Decentralized (Sensor kve) Fusion
Decentralized architectures maintain independent tracks at the sensor level. and generate a single master
track at a higher levei 1211 (Figure 2-2)- The local tracks are updated over al1 measurements; however, to
save on bandwidth and central processing reqhments, somethes ouly a subset of the locally filtered
quantities is passed to the global node, There is no restriction on lateral communication in cases where
feedback is required to produce complete local reports h m dimensionally insufficient sensors. Because
of its topology, decentralued architectures are inherently weU served by track onented &ta association
techniques. Its main benefits are robustness and parallelism (if distributed processing is favoured).
Several variations of optimal and suboptimal decentralized architectures have been pmposed in the
Literature [2 11, [22], which typically trade performance for computational efficiency. Given the premium
on fault tolerance, the decentraüzed architecture was identified as the early favorite for application to the
AVRSS, contingent on its ultimate perfomiaricc against a nval seqyentïal stmcture. Two task spec&
decentralized architecture are developed in Chapter 3 and later simulated in Chapter 5.
SENSORS LOCAL
PROCESSORS
GLO6AL A PROCESSOR
MEASUREMENTS LOCAL ESTIMATES
Figure 2-2 Decentralized (Sensor Level) Fusion 1161
2.4.4.2 Centralùed (Measul~nent Level) Fusion
Whereas in decentralized fùsion separate tracks are maintained at each seosor, with centralized
architecture, the cornpiete set of measurements is reported directly to a single processor (Figure 2-3).
Because all observations are available, centralized h i o n d e s weU with MHT, allowing highly accurate
data association aven extensive processing faciMies. However, it is susceptible to track contamination
should just one of the sensors fail; whereas with decentralized fiisioa, the faulty sensor could be pulled
from the system without affecting the other local tracks [21]. Thus, cenaalued fusion ofiers slight
potential advantages in accuracy that might not k m i k d in praftice.
SENSORS
i =2 GLOBAL k (k l k)
PROCESSOR P~(klk)
MEASUREMENTS Xo(k+l Ik) T Figure 2-3 Centralized (Measurement Level) Fusion
2.4.4.3 Sequential Fusion
Sequential fusion is similar to the centralized architecture in that a single filter is used to process aii
incoming measurernents; however, unlike the latter, it does not requires additional logic to align temporarily
non-cornmensurate observations before fhsing them. Sequential fbsion architectures are inherently
appropriate for multisensor systems because they can accommodate a single state mode1 and multiple
measurement models. Specifically, separate gains are rnaintained for each sensor to pnxluce the a
posteriori estimates. Sequential fiision offers optimal performance but is charactecized by the same
weaknesses as centraiked h ion . As it is the de facto approach [23] [24] for fùsing i b d and radar
data, sequential fusion was given çtrong consideration as for use in the A\LRSS.
2.5 Short List of Design Appmaches
This chapter has motivated the tracking and data h i o n problem as an application of estimation theory.
The practical components of a comprehensive tracking system were presented in order to outline the
choices available in designing the A W S . Based on an initial review of options, a short list of
approaches may be constructeci (Table 2-3)-
1 Candidate Iksim Strotegies 1
1 Architecture: 1 SequenfiuI Fusion 1 DecentraIîzed Fusion 1 Track Initialization:
Data Association:
Sequential Method 1
-
Nearest Neighbour
1 Maneuver compensation: 1 Reinitialuution of Covuriunce MarrLr at Senror M e / 1 Table 2-3 Short List of Design Approaches
Emphasizing the efficiency requirement eliminates iMM and MHT for maneuver compensation and &ta
association respectively, while the need for robustness suggests a decentralized architecture. In this
regard, sequentia l initiahation and Nearest Neighbour assignent are particularly attractive strategies, not
only because they are efficient, but because they represents a logical starting point for a preliminary
design. The case for sequential fùsion &as already been motivated.
Chapter 3
Detailed Evaluation of Candidate Architectures
In Cbapter 2, various design requirements were visited at a descriptive levei, resulting in a short list of
candidate strategies for architecture, and a final choice for data association and mck initiation schernes.
In this chapter, each class of architecture is developed as a set of one or more practical algorithms,
starting with sequential fùsion. Data association and track initiation are then addressed. A final choice
for one approach over the others is made later based on their performance in simulation (Chapter 5).
3.1 Candidate SF-1: Sequential Fusion
As discussed in Chapter 2, the sequential filter offers potentially accurate results and intrinsically handles
asynchronous data rates, but has low fault tolerance 12 11.
3.1.1 State and Measurement W s
In this application, the state comprises of the three-dimensional Cartesian positiodvelocity vector
X W =[x(k ) y(k ) ~ ( k ) W) YW) i(k)IT (3- 1
The system mode1 for the constant velocity target profile is thus given by
X(k + 1) = F ( k ) X ( k ) + w(k) ; w(k) - N ( o , P ( ~ ) ) (3-2)
where
and
Note that
AT = [tk -t& O (3-5)
is the generat time di fference between the two most m e n t measurements which amve at random fiom
-21-
either sensor at times t, and &-,.
The measurement mode1 diffets for radar and imager updates (Figure 3-1). For the radar it is given by
zR(k) = hR(x(k) ,k )+vR(k) ; vR(k) - ~ ( 0 , ~ " ( k ) ) (3-6)
where ZR(k) E d3 includes measured range r(k), bearing û(k), and elevation m). These quantities are
related to the state by the non-iinear observation vector
r = range 8 = bearing 41 = elevation
Figure 3-1 Relationship Between State (Cartesian) and Measurement (Sphencal) Coordinate S ystems
The imager measurement
zl(k) = hl (x (k ) . k)+ v'(L); d ( t ) - N(O.R/ ( k ) )
is similar to Equation 3-6. except that range is not observed. Z1(k) E !Jt2 is explicitly described by:
The measurnent noise is assurned to be uncorniateci both between sample and is given b y
~ " ( k ) = ~ [ v ~ ( k ) ( v ' ( k ) ) ' ] = d i a ~ [ . ~ , , 4. +]
R' ( k ) = ~ [ v ' ( t ) { v ' ( k ) } ~ ] = dio.[c$. $1
3.1.2 Filter Formulation
The sequential filter is formulated as an extended Kalman filter; however separate update equations must
be generated to accommodate the differing measunment models. Referrïng to Table 2-2, the Jacobian
H(k) and Kalman gain K(k), are computed differently, depending on whether the rneasurement cornes
fiom the radar or imager.
Radar Update Equations. In the radar case, the conversion between Cartesian and Sphencal
coordinates is accomplished by evaluating
with respect to the a priori Cartesian state estimate, giving
where r, and r2 (A) represent
and
The gain is thus calculated by
~ ' ( k ) = ~ ( k l k - 1 ) ( ~ ' ( k ) ) ~ [ ~ " ( k ) ~ ( k ~ k - I ) { H " ( ~ ) } ~ + ~ " ( k ) ] - '
where X(klk- 1) and P w 1 ) are the a priori state vector and covariance e m r matrices, respectively . Accordingly, the state is updated by
xR(k1k) = K(klk - 1) + K ~ ( L ) [ z ~ ( R ) - ~ ( ~ ) k ( k l k - LI] (3-lf)
andcovarïanceby
P(kl k ) = [I - ~ ~ ( k [ k ) ~ ) ~ " ( k ) ] ~ ( k l k - 1) (3- 1 8)
Imager Update Equations. The equivalent update equations for an imager measurement are developed
in the same approach- We have,
~ ' ( k l k ) = k(k(tlk - 1) + ~ ' ( k ) [ ~ ' ( k ) - ?i(k)k(klk - 1)]
P(k(k) = [I - ~ ' ( k l k ) f f ' ( k ) ] ~ ( k l k - 1)
Radar and Imager Prediction Equatioas. Because the arriva1 time of the next measurement is not
known, the a priori estimates of the state and e m r covariance matrix must be calculated before the
measurement update at t h e & ; i-e,
X ( k l k - 1) = ~ ( k ) k ( k - llk - 1) (3-24)
Although Equations 3-24 and 3-25 are calculated in the same manner as the 'prediction'equations of Table
2-2, the latter form ernphasizes the required physical structure of the present filter, and avoids the
mistaken assumption that AT is constant. Thus, as soon as the system receives a measurement h m
either radar or imager at time k, it calculates the a priori estimate for time (klk-i) imrnediately followed
by the a posteriori estimate for the (klk) index. The filter's timing is depicted in Figure 3-2, with emphasis
on the gain that must be calculated separately depeadiag on whether the m e a m m e n t came from the
radar or imager.
Figure 3-2 Sequential Architecture for A L S S
3.1.3 Remaiks
The sequential fomulation has received favor in case studies of highly dedicated active/ passive aackers.
For instance. Romine and Kamen [24] developed a similar filter which includes additional States to
account for the di fference between the target 's centre of reflection (which the radar observes) and its
geomenic centre of mass (wbich the imager observes). A simulation of the sequential filter detailed in
Section 3.1.2 is performed in Chapter 5 for two cases: (i) equal update rates across sensors, and (ii)
unequal update rates.
3.2 Decentralized Architectures
Decentralized h i o n is more robust than sequential fiision as its maintains separate local tracks. It may
be implemented in optimal and s u b - o p h l configurations. In this thesis, two suboptimai approaches are
considered as well as one optimal design. They are, in order: Static Fusion (DF-l), Static Fusion with
Feedback (DF-2), and Dynamic Decentralized Fusion @F-3).
3.2.1 Candidate DF-1 : &tic Deceirtrslized Fusion
3.2.1 .l Background 121
Static fusion combines the locally produced üwar minimum variance 0 estimates %(kl k) for al1
A A
i = 1 .. p available sensors to pioduces a global estimate, X , (klk) defmed by
The estimation error is minimized according to the performance index
where Mi is an unknown constant weight applied to each local estimate, and Xe) is the tme state. The
result is a h e d estimate that minimizes the square of the estimation emr,
e,(k) = X(k) - ~~k~(k(k) (3-28)
which is assurned to be orthogonal. Obeying this criterion produces an estirnate for the fonn:
which cm be recast into the ' information' form:
where, in general,
is defined as the 'pseudoestimate'. The information fom is advantageous as it can reduce the number of
required matrix inversions.
Static h i o n is the least computationally demanding decentraiized technique because only the most recent
local estimates are used to form the global estimate-, and because information is only pessed in one
direction (local to global level). However, because it does not condition the local estimates with global
information, it is less accurate than the 'dynamic' fùsion detailed in Section 32.3- An improvement c m
also be achieved by feeding back the global information to the local nodes to produce 'static fusion with
feedback', as in Section 3.2.2- Before exploring the latter options, a static formulation is developed,
3.2.1 9 Application of Static Furbn to the AURSS
In the present context, the goal is to produce a complete global estimate (E S3) h m local imager (T) and
radar (R) tracks (E 3t2 and E 3t3 respectively), The h i o n must be performed in spherical coordinates
(denoted by subscnpt S) in order to decouple the cornmon quantities (bearing, elevation) fiom range,
which cannot be estimated by the imager- It is assumed that each sensor bas made available the
following pseudoestimates:
and
To take advantage of the constant velocity assumption, an extended Kalman filter with Cartesian state
mode1 is employed at the radar level. The resulting estimates and P matrix must then be transformed to
Spherical coordinates using Equations 3-7.
Bearing and elevation estimates may be produced at the FUR level using the linear image Plane filter- A
complete point for Fusion can then be f o m d by assigaing an arbitrary range. to augment the imager's
state to S3 (range, bearing, elevation) [20]; Le.,
Because F is arbitrary it should be assigned no confidence (it is only incIuded to produce a dimensionally
compatible point); thus, the correspondhg entries of the inverse covariance matrix for the imager estimate
are set to zero, leading to
[ i y ( k l k ) i t =
The two local estimates c m then be fused in information form. The global quanttities are then gïven by:
d,"(klk) = <îS(klk) +&(klk) (3-37)
and
[ e j G ( W r L = [p:(klk)r' + [ ~ ~ ( k l k ) i ' (3-3 8)
3.2.1.3 Gemrating Local Psmdoedtnates
In the preceding development, it was assuned that appropriate locat pseudoestimates are produced by
both sensors; specifically, that the radar filter provides a three dimensional estimate in Cartesian
coordinates that may be converted to range, beanng and elevation, and that the FLIR filter p d u c e s a
corresponding estimate of bearing and elevation. The production of the radar estimate via the extended
Kalman fiIter bas already been discussed. Two appmaches to the F L R filter are now considered (Case
1 and Case 2).
Case 1 : Linear image Plane F i h . The most efficient strategy is to model the target's centre of m a s as
a single pixel that negotiates a constant velocity course in the i-/' image plane (Figure 3-3). This allows the
DKF to be employed to f d l effecf resulting in a very efficient formulation. The conversion between pixel
coordinates and angular coordinates is achieved by the straightforward method presented in Section 4.1-
Bar Shalom demonstrates good results using a single image plane filter to track a target whose dyoamics
are unluiown [25]; however, in this case the given systern model (constant velocity in Cartesian
coordinates) does not infer h e a r kinematics in the image plane (which implies constant angular rates);
thus, a mismatch between the local filter state models is incorporated (Figure 3-4). Despite this inherent
flaw, the use of the linear image plane filter was not ruled out in case the mismatch pmved to have a
negligible eflect.
j [pixels]
Figure 3-3 Image Plane Coordinate System
B!!l
Target Centoid
Figure 3-4 Mismatch Between Lînear Cartesian Dynamics and Linear Image Plane D ynamics
2: IlodiMd-Polar FP$r m. Second, the Modified-Polar (M-P) filter is considered, which pmvides
an exact mapping between linear Cartesian dynamics and Polar1 dynamics, thus ehinating the mode1
mismatch. The Modified-Polar filter differs h m the standard polar filter in that the inverse of range is
estimated (instead of range); defuiing the state this way (see Y(t) below) decouples the observable
components of the state vector (bearing rate, range rate over range, bearing) from the unobservable
component (range) assuming chat range cannot be measured,
The Modified-Polar (MF) filter refonnulates the constant Cartesian velocity problem using an EKF with a
non-linear state mode1 and iinear measurement modet The state vector, denoted by Y(t) in this case, is
given by
[rad I s]
The non-linear differential equations describing the evolution of the target course (constant veloçity in tbe
Cartesian plane) are given by:
To estimate this system with the EKF requires that Equation 3-40 be integrable in closed fom. While an
advantageous mapping between Cartesian and Polar spaces makes this possible, the resulting filter
includes a highly non linear state Jacobian, which unfortunately, proves too cumbersome to justiQ using
this formulation.
3.2.1.4 Rernarks
A simulation of Candidate DF-1 (Linear image plane filter) is performed in Chapter 5 to gauge the
' The M-P filter is discussed in terrns of generating an estimate for range and bearing. The elevation component could be estimated with a second filter.
effectiveness of the static appmch, in spite of the mode1 mismatch. Static hision using the Modifïed
Polar filter was discounted as an option due to the sigaificant computational requirements. (a fact, using a
dynamic technique to circwnvent the observability issue promises both more accurate results and less
costly implementation. Before the dynamic approach is detailed, another initial candidate is presented:
Static Fusion with Feedback.
3.29 Candidate DF-2: Static Fusion with Feedback [16]
In the very early stages of this project, Static fùsion with feedback was identified as a strong contender
for the final architecture. The scheme is attractive because it genetates an optimal estimate if the global
processor is slewed to the local rate, and demands only a slight increase in computation with respect to
static fusion. Accuracy is increased by feeding back the global track information to the local level, where
it is used to condition sensor level-estimates. The pseudoestimate equations conesponding to the process
as depicted in Figure 3-5 are [2 11:
d, (kfk) = ~-'(k)&(k - Ilk - I ) + H,(k)R;'(k)Z, (k) ( 3 4 1 )
and
where
Provided the global update rate matches the slowest local filter update rate, static fùsion with feedback is
mathematically equivalent to fidl dynamic h i o n [163; however, it saves oa computational requirements for
given state and measurement equations. Unfortunately, closer analysis reveals that this configuration is
impractical given the dimension of the FLIR level estimates. Specifically, as seen in Equation 3-41, the
technique requires a cornmon state model in order to advance the global estimate. The only way to
accommodate this constraint is to employ two modified polar filters, or to recast the dynamic model to
represent a constant angle rate (which is not a practical assumption). As an architecture based on the
MP filter has already been nileci out, the static fiision with feedback is eliminated as a design option prior
to simulation, leaving fidl dynamic firsioa as the remaining decentralized contender.
SENSORS LOCAL
PROCESSORS
- 1 GLOBAL PROCESSOR
LOCAL MEASUREMENTS PSEUDOESTIMATES
Figure 3-5 Static Fusion with Feedback [20]
3.2.3 Candidate DF-3: Fully Oecenfralized Dynamic Fusion
Fuily decentralized fusion provides the optimal LMV estimate for the global mean. Stated othenivise, the
formulation represents the theoretical best option in tenns of performance. The equations for
decentralized fusion may be developed in either covariance or information fom; however. the latter is
more appropriate for practical implementation (see Section 3.3). The global a priori pseudoestimate and
inverse covariance matrix are defined by [ 161: N
d , (k + 11 k) = f-'(k)&(kl k - L) + x [ d l (k + 11 k ) - F - ~ ( ~ ) J , (kl k - l)] irl
and
Inspecting Equation 3-45 reveals not only that the global and local states vectors must comprise the same
quantities, but aiso that the states must be updated by a commoa transition matrix. These requirernents
are circurnvented by employing two similarly configured extended Kalman filters to process both radar
and image data To create a complete estimate in the second case, a range terni is assigned to the imager
angles. Instead of using an arbitxary value with infinite variance (as in the static case), the wist ment
radar measurement is fed into the image fdter, and propagated to the required time for global processing.
SENSORS LOCAL
PROCESSORS
- - - R = Range 8 = Bearing LOCAL C#I = Elevation PSEUDOESTIMATES
(C
U k ) di(klk) I
- - -
Figure 3-6 Dynamic Decentralized Fusion
9 4 1 pl(klk) radar
This method could have also been used in part to accomplish sîatic h i o n with feedback; however, it is
desired to minirnize the amount of feedback in the architecture to avoid the dissemination of corrupt data
should one or both sensors fail, A single unidirectional transfer is required by the present configuration
(radar to imager). On the other han& the design of a static system with feedback would also cequire the
retum of the global estimate to the local fiiters.
*
d4kM
3.2.3.1 Timing ConsWraticm
Until now, the candidate filter formulations have been presented under the assumption that each sensor
provides its estimate to the global prccessor at a unifom rate. in this case, the assumption is untenable.
While the imager updates at IO h m e s per second (a constant), the radar reports only when the target
appears in its line-of sight. Because the radar antenna rotates back and forth between two arcs at a
moderate rate at best, the time between intersections wiU be govemed by a nonlinear hc t ion and almost
certainly be non-uniform. To deal with the unsynchronized measurement schedule, a stight modification to
the fusion architecture is necessary. SpecEcaiiy, Equatious 3 4 5 and 3-46 mua be recast in a more
general fonn. Letting the a posterion time indices (tJW k fepresented simply as (W. the global
pseudoestimates can be fonned by p l :
and
where t, and t, are the indices o f the most recent global and local estimates respectively. It is assumed
that t , 5 t , < t,,+,,i As depicted in Figure 36. the additional ternis in Equatioas 3 4 7 and 3-48
propagate the local estimates generated by the radar (sensor i = 1) and the imager (sensor i = 2) to the
global update tirne. The time-aligneci local estimates are given by
, ( t k t ) = ~ - r ( t , t ) , ( t t ) i = 1 , 2 (349)
and
( t ) = F r ( t k t ) l ( t t ) ' ( t k , t ) : i = 1, 2 (3 -50)
fk- f tk tk+l - y---- f -*-- ----- *--b GLOBAL
Time i tkl : f(k+l) 1 : -
[SI . . . . . . . . radar . . I-)i i =1 . . . .
f(k-1)2 lk2 f(k+1)2 -- e---.-- -- -p - - -0- . . . . imager . - . . . . . . . . . . i = 2 . . . . A
dt(tki 1 hi) dr(k 1 bci) Note: TFLIR = TG = const TRAOAR = variable
Figure 3-7 Time Aligment at Local And Global LeveIs
-34-
Candidate DF-3 is simulated for 2 cases in Cbapter 5: (i) equal imager/ radar update rates and (ii) unequa1
update rates. Having eliminated Static Fusion with Feedback OF-2) eatly h m consideration, DF-3 joins
sequential (SF-1) and static decentralized @F-1) options as one of three possibilities for the AURSS,
3.3 Replacing Stanôarâ EKF with More Efficient Formubtion
In Section 2.2, the extended Kalman filter (EKF) was fonnulated for use in this application as the
minimum variance estimator. While the EKF is essentially optimai, it is not necessariiy computationally
efficient. In this case, slight modifications to its standard form can be made which afford a significant
computational savings with a negIigible sacrifice ofperformance. The drawback to fomulating the state
equation in Cartesian coordinates is the need for a non-linear measurement mode1 (Equation 3-52), with
which to process the sphe~cal observations. However, it is possible to replace the standard measurement
equation by a linear fùnction (Equation 3-53), afier transferring the nonlinearities to the e m r tenn. The
reformulation allows the gain matrix to be calculated in sphencal coordinates; and only later rotated into
Cartesian space for the state update. The resulthg 'Modified-Baheti filter' [13] (Table 3-1) is at least
twice as efficient as the standard EKF [13]. Note that the formulation employs a rotation matrix
(Equation 3-65) to map the emor covariance matrix between sphencal and Cartesian coordinate systems.
This key approximation assumes the offdiagonal terms (range rates / angular rates) are minimal, a valid
assumption given the present target dynarnics.
3.4 Data Association and Track Initiation
The objective of this thesis is to design an efficient, robust and effective tracking system that best exploits
the existing facilities aboard the LAV Recce. Accordingly, al1 subsidiary systems proposed for the
AlLRSS must not only be diable, but also cost effective. The issues of data association and track
initiation are now addressed according to this philosophy.
3,s Data Association
In a multi-target and/ or cluttered environment, a technique is needed to associate al1 incoming
measurements (across ail sensors) to their respective sowces. As outlined in Chapter 2, &ta association
can be accomplished via Multiple Hypothesis Tracking 0, Probabilistid Joint Probabiiistic
System mode1
Measurement mode1
Prediction
Correction
Information representation
Definitions
*(OP) = initial estimate of MP state vcctor
S-' (OP) = initial estimate of inverse spherïcal enor covariance nratnx
Hcn =[f, O,]
R= = F,R=F ,~
c E cartesian quantity s i spherical quantity
Table 3-1 Modi fied-Baheti Filter
Data Association (PD& JPDA), or by Nearest Neighbow (NN). While optimal association is achieved
by MHT, the approach requires an inordinate amount of pmessing and memory cesouces to generate
and maintain its hypotheses. The JPDA represents a more attractive approach, although it also demands
a signifrcant computational overhead, The Nearest Neighbour technique, on the other hand is both
eff~cient, and cm dso be very effective pmvided the target count and false alann rate are not too hi&
In the present case, the NN is a logical choice because: (i) at present the LAV Recce has no data
association facility, wbatsoever, and it is reasonable to approach an upgrade by methodically increasing
the prototype's complexity until the desired performance/ cost ratio is ceacheci; and (ii) there is no
guarantee that a more cornplicated technique will yield significantly better results. Furthemore, because
the radar and the image tracker are developed concurrentiy in this project, the need for a more
complicated technique cannot be established at this point,
3.5.1 Nearest Naighbour ûata Association
Nearest Neighbour (NN) data association makes irreversible 1 : 1 assignmeats of current measurements to
existing tracks by detennining - statistically speaking - the most likely combination of these entities. To
formulate the assignment problem as a practicaf algorithm, sets of vaiidation and association matrices are
used to detemiae fmt the pemissible combinations, then their respective Likelihoods. Let the set of al1
measurements recorded at time k be designated as
E {~,(k),~(k),-..,z.~(k)) (3-7 1)
For a measurement to be considered for association with one of T established targets, it must fall within
a specified distance of the target's predicted location, &k(k - 1). The maximum pemissible distance is
set by a gate G,. according to a chosen confidence criterion. A validation matrix can then be established
to record the possible assignments. Forj = 1 .. Mmeasurements ernanating fiom t = 1 ., Ttargets, the
validation matrix is defined by
~ , ( k ) = [ ~ , ( k ) ] c ~ ~ , j= 1 ,..., M t = l , . - - , ~
w, in Equation 3-72 is a scalar fùnction indicating simply whether measurement j
(3-73)
falls within track gate t.
Once validation (thresholding) bas been penormed, an 'assignment' matrk is made by repiacing the
binary decision with a distance hction that indicaîes how clooely the latest measurements and predictions
coincide. The NIS (Section 2.2.1) provides a convenient measure because it is a n o d i e c i quantity.
Using Equatioa 2-19 to calculaîe the NIS for each potentid measummentl track pair re!suits in the
following ma&:
where j t denotes the aforementioned pairing. As an example. consider the scenario Figure 3-8 [13].
O = validation gate A = measurernent X = propagated track E = normalized distance
Figure 3-8 Samp Ie Nearest Neighbour Data Association Scenario [ 131
According to the preceding development, the validation and assignment matrices are (Tables 3-2 and 3-3):
Table 3-2 Validation Matrix for Hypothetical Tracking Scenario 1131
1 - denotes an inadmissible assignrnent 1
Table 3-3 Assignment Matrix for Hypothetical Tracking Scenario [I3]
Once the association matrix is formed, the measurement-to-track assignment is performed by finding the
pairs (k) such that their cumulative distance x $ ( k ) is minimized. The standard solution r
assumes that a decision is made after every update. The NN is constrained to maximize the total number
of associations to reduce the candidate solution set. In the simple case, the algorithm would fom the
following pairs: (X, , A,), (X, , A ,). Many algorithms have been developed to mechanize NN; for
instance the Munkres algorithm [27] is an appropriate realization.
3.5.2 Application of NN Data Association to Dacentralired Fusion
Should a decentralized architecture be chosen, data association would have to be performed at both the
local (sensor) level and at the global (fùsion) level. Association at the local level concerns the assignment
of measurements to tracks. This problem is straight forward in the case of the radar filter, and consists of
implementing the routine described in the previous section. In case of the imager, association is slightly
more involved because the filter is required to fonn a complete position estimate using angle
measurements from the imaging sensor and a range estimate fiom the radar. Thus, association is first
needed to match the latest imager measurement to an existing radar (range, bearing, elevation) estimate-
Only then can the 'complete' point be associated with previous imager tracks. It is important to keep in
mind that al1 initiations are perfonned by the radar tracker; thus there is no dilemma in processing
extraneous imager detections, even if they originate fiom a new target; i.e., by default, new tracks will not
be initiated if the signal source is seen only by the imager.
Data association at the global level is accomplished by applying the NN method described in Section
2.4.3.2, except that tracks are associated to tracks instead of measurements- To account for the total
uncertainty, the association could be perfomed over the intersection between the prediction gates, one
centered around the radar estirnate, the other around the imager estimate. A more convenient (but
equivalent) approach to account for the total uncertainty is to increase the size of one of the two gates in
proportion to the size of other's covariance. Using tbis strategy, it becomes possible to use the same
algorithm for assoçiating at global and local levek-
3.5.3 Application of NN M a 1i4rtociatiin to Seqmtial Fusion
One advantage of sequential fusion over decentralized fission concems the complexity of the attendant
data association. in the sequential case, it need only be perfonned at one level; Le,, measurement-to
track, and can be accomplished in either complete or incompfete dimensions, depending on whether the
latest measurement came h m the radar or the imager. In either case, the NN algorithm as previously
described would be used.
3.6 Track Initiation
The requirement for an explicit track initiation scheme depends on the attendant choice of data
association- MHT based tracking systems accomplish the task automatically, whereas RDA, PDA, and
NN do not. In general, track initialization is performed either by batch means or sequentially, Batch track
initialization schemes require a complete set of detections to be saved over time in order to make an
optimal decision; however, the present application cannot afford this Iuxury. Sequential methods, on the
other hand, allow decisions to be made according to the last few observations. Of the sequential
algorithms, the 'Logic Based Muititarget Track Initialization 1281 is particulariy suitable.
Logic based multitarget track initialization uses knowledge of the assumed target dynamics and noise
statistics. ï h e test begins by considering al1 m measurements collected at time k. Denote these by set Zi,
i = l..m. For simplicity, Zi is divided in to one of its f =I..n dimensions: Zi' . The test begins by establishing
an acceptance region (Le., a one dimension gate) around each measurement, say a bearing. The size of
the acceptance region is set according to a desired confidence level. The gate's location is then
propagated over the sample p e n d and associated with the next detections (falling within its limits). To
account for al1 possible speed profiles, the gate is increased in size according to the maximum and
minimum possible velocity components of the target along the chosen direction. The process is continued
for a set number of iterations. If a measurement falls within the gate over the prescribed number of
updates. then the previously tentative track will be initiateci.
According to the preceding description, the maximum difference between two measwements recorded at
times L;Cb and t=t,, is 1291,
d: - max[ (+) -&k - 1) -PLAT) ,O] + -[ (+) -&k - 1) -&,,AT ) ,O 1 (3-75)
where AT represents the time between the measurements, and where v,, and v,, represent the
maximum and minimum possible components of the target's velocity. Taking the variance of the
measurement noise into consideration, the normalized statistical difference between the detections is:
Qï -d,$& (k - 1) + R, (k)rld, (3-76)
Assuming the innovation process to be Gaussian (zero-mean), the detections should be associated if, for a
given confidence leveL the standard NIS test statistic is satisfied; i.e. if Du 2 y . where y is a cbi-
squared distributed threshold for a system with n degrees of fieedom. The process is swnmarized by the
following steps in [29]:
Step 1: Starting with a detection fiom scan k-1, establish a Pte using Equations 3-75. For every
measurement from scan k falling within the gate, set up a potential track.
Step 2: For every potential track (consisting of two measurements), perform a straight line extrapolation
to the third sampling instant- The size of the acceptance region for the third scan is determined by the
variance for the estimate in the given dimension (calculated using Equation 3-76)
Step 3: Drop potential tracks whose gates do not see a detection at the third scan. Spilt the track if
more that one measurement is recorded.
Step 4: Continue validation process for desired number of scans. Confkm remaining tracks and
commence tracking in full dimension.
Chapter 4 lnfrared Image Processing
Up to this point is bas been assumed that the LAV Recce Surveillance Suite (LRSS) provides angular
measurernents of target position, characterked by knom noise statistics; however, this is not the case-
The information generated by the Westinghouse MicroFLJR is an analogue representation of the entire
surveillance volume (targets plus background). Consequently, a method must be devised to extract the
required kinematic information. The appropriate mapphg may be devebped given the Iocation of the
target's centre of mass in the image plane, as well as the sensor's horizontal and vertical fields of view
(FOV), and the size of the image screen in pixels. The compIete procedure involves (i) estimating the
target's mean position (centroid) and variance; anâ (ii) transforming this pixel position to an equivalent
angle representation. These requirements will now be detailed, beginning with the second task-
4.1 Mapping Between Pixel and Angk Coordinat8 syriems
Let the target's image location be specified by the followhg parameters (Figure 4-1) 1241:
('w % 1 Vertical and horizontal FOV [radians];
WC, A<pC ) Instantaneous azimuth / elevation of an arbitrary point (i-e., the target centroid)
measwed wxt. the center of the image thme [radians];
{MM, NN} Total vertical and horizontal dimension of image screen Cpixels];
{pi 3 pj) Vertical and horizontal screen position measured w.r.t, to top lefi haad corner,
i = I . ... . MM, j j= 1. ... . NN Ipixels];
{ci , cjI Vertical and horizontal screen position measured w.r.t- screen centre [pixels];
{u,. u d Vertical and horizontal angular extent of each puel [radian / pixel].
j [pixels] --*
n [pixels]
(m. n) : (1.1)
Figure 4-1 Target Location in the Image Frame
The target centroid is typically repocted as the point {pi , pi}, although it is more convenient to work Mth
{ci , ci}. It is possible to map between these coordinates via the following one-to-one transfomi:
c, = 'Iy - pi verticaL location
c, = p, -y horizontal location
nie angular quantities {AW, Aqf may then be determined directly h m {ci , cjJ via
(4-1 ab)
where the pair {u, , h) is detemhed by dividing the total FOV by NN and MM respectively. In this case
a FOV is 200 x 150 milliradians is represented on a 640 x 480 pixel plane. Detenniaing the instmtaneous
field of view (IFOV) in this manner neglects lens aberration; which is a ceasonable assumption provideci
the lens's focal Iength,f, is large compared to the horizontal and vertical dimensions of the colfecting
element. Typically this is the case for a scanning device like the MicmFLIR Note that in this treatment
sub-pixel accuracy is not essential and for that reason the size of the FLIR's blur spot [30] is not
considered (each pixel spans a vertical and horizontal field of view of 0.3 125 radians in each direction),
IC = range of tmget csnlroid & = bang (LOS) BC = bearing (cmtroid) cpr = efevsaiarr (LOS) Qic = devation (csritroid) A = local quantity
w.rL COS
Figure 4-2 Relationship Between Local and Global Coordioate Frames (24 ]
Once the target's angular location has k e n detemiined with respect to the h e center. an equivalent
position. {Oc. qP 1, can be established in ternis of an arbitrary coordinate frame via the augmented rotation
mapping as outlined in Amex B [3 11; i-e.,
rotation
{A@., ~ f ) mappiw
In practice, this operation is required if the global fusion space does not coincide with the sensor's
measurement space (Figure 4-2); specifically, if the sensor's LOS (8, , cp, ) with respect to the arbitrary
h e is non-zero.
4.2 Extracting the Twget Centmid fmm a Noisy Image
Section 4.1 establishes a mapping to relate target image position {pi , pj j to the equivalent angular
representation {BC, qf}. This section is concerned with constructing (pi . pi) in the fmt place. in general.
the problem may be attacked in one of two ways: by cousidering the evolution of the image over tirne
(multiframe image analysis), or by considering the attributes of a single captured image (single thme
image analysis). In both cases, the images are pcovideâ by an analogue to digital converter (i-e-,
framegrabber). Note that the FLIR is a scanning device, and as such the entire search volume will be
scanned in sections by a rosette onto either a singie detector element, or an anay of elements. The h U
search volume is covered at a sufficiently high rate tbat, for practical purposes, a digital rendition of the
entire image over one scan can be assumed to be produced instantaneously. Because the image capture
rate (proposed at 10 thmes per second) is slow compared to the refkesh rate (in the order of 100 fiames
per second), the Nyquist sampling criterion will be satisfied.
Multi-frame image analysis involves either subtracting consecutive fiames tiom each other or correlating
them [32] to reveal movement. The technique is versatile in that it applies equally well to both visual and
infrared imagery. Could the camera be rendered completely motionless, this would be an attractive
strategy - provided that the target were the only moving object. However, in an operational environment
the wind has a tendency to cause the sensor platform to vibrate, creating the illusion of movement in the
search volume; fucthermore, trees and non-stationary objects in the background will appear as possible
targets if their motion is detected.
Conversely, with single fiame image analysis the object is to isolate the target based on the characteristics
of a single image. in the infrared case, an obvious strategy is to exploit the difference between the target
and the background on the basis of their electromagnetic radiation profiles. Specifically, assume that the
captured fiame is recorded as a 256 level grayscale image. The value (level) of each picture element
(pixel) of the MM x NN pixel matrix will be proportional to the irradiance (H) [33] (fkom target and/ or
background) seen by the detector element corresponding to the given pixel.
Based on the preceding discussion and on the precedent set in the literanire, the single h m e method
[22],[25], [34] is chosen for use in the AILRSS.
4.3 lnterpreting the Anaîogue Vidco Signal
Single frame image analysis involves separating the target h m the background by exploiting the
difference in their radiation profiles. To achieve a desired probability of detection (PD) and probability of
false alarrn (P,J, a minimum SNR must be achieved. For an inf?ared imager, the SNR is given by 1331:
where Signal voltage change going h m background to target:
S ystem noise (accounts for Johnson, shot, flicker and signal noise);
Constant terni (accounts for FOV, area of collecter, area of detector, and system
fiequency bandwidth);
h w e r and upper Iimits of infiared bandpass (Westinghouse MicroFLïR operates
between 8 - 14 pm);
Target radiance (power emitted by a unit area of source into a soiid angle);
Background radiance;
Atmospheric transmittance, T, < 1 (accounts for signal loss due to atmospheric
scattering and absorption);
Optical transmittance, to < 1 ( accounts for signal loss due to passage of signal
through optical elements); and where,
'Dee-stai (a normalized measure of a sensor's detectivity as a function of
wavelength).
The terms in Equation 4-4 cm be divided conveniently between those which depend solely on the system,
and those which depend solely on the operating conditions. The latter group (r, N,,, NLB) accounts for
the effects of weather and targetl background radiation profiles, which in turn are determined by the
range of observation, and the difference between target and background temperatures. This classification
is use hl, as it establishes a rapid and simple means of calculating the system's performance envelope for
a required Pm and PD.
A more complete analysis of systern perfonnance is not possible within the context of this project, as most
of the imager's technical specifications are propriety information. However, should the proposed design
be implemented, it would be valuable to establish the Receiver Operating Characteristic (ROC) curves,
which provide an explicit relationship between Pm, PD and SNR.
4.4 Single Frame Estimation [34): m i n i n g Cenüoid üœn and Variance
For the purposes of this investigation, it is assumed that the analogue signal is sampled at a constant rate
( 10 frames pre second) to produce a sequence of 256 level grayscale images. Single fiame estimation
invoIves (i) applying an intensity bandpass to the original sampled images; (ii) quantizing the remahhg
non-zero pixels in each image , aad (iii) clustering the resultant sparse distribution o f binary points into
meaningfb t groups (targets) to eLiminate unlikely candidate clusters.
4.4.1 Background
Because the target is assumed to be an operating vehicle, it's radiant emittance (W) (Wcm-') should be
greater than the that of the background. This information is represented by the value of each pixel in the
MM x NN image fiame in terms of a 'grayscale intensity,' 11, which is proportional to the inadiance (H)
of the source in each pixel's fieid-of-view. In the case where the background inrdiance is negligible, 1, is
given by
where Si is a random variable representing the grayscale inteasity of a pixel within the target extent, T.
and where i and j represent the location of an arbiaary point in vertical and horizontal pixels . The
location of the target's centroid (with respect to the top lefi hand corner of the image h e is given by
the point ( p i pj), where
Mhf NN
CClv(k)j (4-6 ab)
i-i j-1 i-1 j-1
Dropping the dependence on k for notational convenience, and without making any assumptions
conceming the distribution of S,, the variance of the target centmid specitied by Equation 4-6 is given by
where p (4) is the mean value of pixel (i). and w h r e CI$ is its variance-
4.4.2 Application of a Bandpass Riter
In this thesis it is desired to characîerize every pixel as either belonghg to the target (Le., i&T) or aot
belonging to the target (Le., i JET). This binary ciassification (quantizing) is accomplished by applying a
bandpass fiiter to the originai image. As a result, the vaiue of each pixel is given by
where 1, and 1, specifjr lower and upper threshold intensities, which are chosen to guarantee a minimum
probability of detecting the target, p, (Figure 4-4). For instance, assuming the target pixeb are Gaussian
random variables with means as pi and variances di establishes the probability of detecting a pixel as
[34]:
Note that the Gaussian assumption is not necessary but is used hem for illustrative putposes.
Figure 4-3 Application of a Bandpass Filter
-48-
The pmbability of detecting a pixel in the bandpass is chus
where the symbol n: has been chosen to represent probabiiity in lieu ofp to avoid coafiising the symbol
with pixel position, p,. Using Equation 4-10, the overall variance of the centmid location calculated witb
respect to (pi, pi) is given by
while the centroid will be located at
where xij(l-ir,) is the variance of an arbitraxy pixel in the target extent. Although the target pixels will not
be independent and identicaliy distributed (iid), in practice they may be treated as such, Under this
assumption, it follows that for mean p(aj) = z and variance &(pu) = n( 1 -IL),
where N, is the total number of pixels in the target extent and where i, and j, represent the location of a
target pixel in vertical and horizontal directions.
Up to this point, the effect of the background has not been taken into consideration. To accommodate
background, the grayscale intensity of each pixel can be described by
PIXEL INTENSITY P ij = O
Figure 4 4 Application of a Bandpass Intensity Fiiter Given Non-Zero Background
where T represents the target extent (composed of N, pixels) and where B represents the background
extent (composed of Nv pixets). It is assumed that the background pixel intensity distribution will lx
spread out over the bandpass (Figure 4-4). As a result, the intensity filter will not be abie to isolate the
target without also detecting background pixels (Le., noise). Thus, the choice of threshold must take into
account not only the desired probability of detectioa h, but aiso the acceptable probability of false alarm,
p,. These tenns may be computed after identwng the rcquirrd moments of the target and background
distributions (fiom a sample image), and integrating theû respective pmbability density fùnctious between
1, and 1,. The presence o f the background pixels will increase the variance of the centroid estimate.
Assuming both target and background pixels are i.i.d. gives
where q and x,, are the probabilities of detecting target and background pixels respectively, and where i,
and i, are the i and j coordinates of pixels in T and N.
A M e r improvement in the centroid estimation process is achieved by grouping the remaining non-zero
pixels into independent 'clusters' on the basis of proximity. By doing îhis, it becomes possible to eliminate
fiom consideration spatial distributions which are too sparse to represent targets. The clusters are
'grown', starting with an arbitrary pixel, and by linking any candidate which falls within a given proximity
distance, d, to the cluster. This sosalled single linkage algorithm [35] is appropriate as it may be
performed in real-the. As shown in 1321, the proxirnity distance should be chosen such that it falls within
the average distance between neighboring pixels in the target (d ,) and the in background (d .). These
quantities are approximately given by
and
where p, andp,. represent the probability of detecting a target or background pixel, as discussed in the
previous section. Bar Shalom [25] determined by simulation that the 'optimal' proximity distance, dP9,
corresponds to the mean of this interval; i.e.,
Clustering with dpo links most of the target pixels while isolating most background pixels. If the
approximate size of the target (in pixels) can be anticipated, then clusters that are 'too small' may be
eliminated, while the centroid location and variance of the remaining clusters can be calculated by
Equations 4-6 and 4-15.
Chapter 5 Simulation of Design Alternatives
The candidate architectures d i s c d in Chapter 3 were simulated by Monte Carlo trials in order to
establish the most advantageous configuration. The figure of ment throughout is the mot meaa square
error (RMSE). Where appropriate, the expected error (EE) and the measurement e m r (ME) are also
plotted. The EE for a given element of the state vector (i.e., position in the 'x* direction) corresponds to
the square root of the filter's covariance matrix for the appropriate element. The overall expected
position error at each point in time, k, is calculated as
E E , i k ) = Jq,<k> + pz(&) + P,(k) (5-1)
where Pii(k) corresponds the EE (squared) of the ith element of the state vector X(k), i = 1 ..6, given that
X ( k ) = [ d k ) y(k) r (k) i ( k ) Q(k) i (k ) IT (5-2)
The RMS Error is calculated in similar fashion:
R M S E ( ~ ~ ( M ~ ) ) = ~ M S E ( ~ ( ~ ~ I ~ ) ) + MSE(~(&I~))+ MSE(~(~I~)) (5-3)
The measurement e m r corresponds to the standad deviation of the radar measurements. It is included
to indicate the besr performance of the current system, which does not have any estimation capability.
5.1 Scenario
The simulation scenario is based directly on a surveillance recording made by the U V Recce. The
target travels along a straight road between two positions at a velocity of 15.8 meters per second. In
Cartesian (X-Y-Z) coordinates, the start comsponds to (663, 1278,431 meters and the finish to (677.
1 3 30, -20) meters, both measured with respect to the observer location. The radar and FLiR are
coIIocated with overlapping arcs. Figures 5-l(a) and 5-l(b) illustrate the course profile as it appears to
each sensor. The imager reports the centroid position in pixel coordinates, assuming a measurement noise
standard deviation of 10 pixels both in i and j directions. The FOV subtends horizontal and vertical arcs of
200 x 150 milliradians, which corresponds to 640 x 480 pixels in image fiame. Thus, u, and y,, both equal
3.125 x 10J radians per pixel. The radar reports range, be-g and elevation assuming standard
deviations of 24 meters, 0.01047 radians and 0.01047 radians respectively. Al1 noise statistics are realistic,
but do not necessarily correspond to the exact specifications of the LRSS. In order to measure the best
possible performance of each estimation scheme, no state noise is included in each scenario, the initial
conditions correspond to the true state values. With respect to timing, two cases are considered: In Case
1, both the imager and radar provide updates every 0.1 seconds. In Case 2, the imager reports every 0.1
seconds, while the radar updates every 1.0 seconds. The simufation is dormes i over 10 seconds
(simulated time). which corresponds to the maximum period during wbich the target is expected to remain
visible in an operational environment.
Target profile in image piane a , q ,
Figure 5-1 Simulated Course Profile for (a) imager and (b) Radar
5.2 Candidate SF-1: Sequential Furion (Case 1 and Case 2)
Candidate SF-1 corresponds to the sequential filter detailed in Section 3.1.
Case 1: Case 1 establishes the e d y standard for fiiter perfoxmance (Figure 5-2). The RMS error for
total position reaches a minimum value of 5 [ml, a six fold improvement compareci to the present system
(indicated by the radar's ME), ï h e benefit of including the imager angles is dernonstrateci in a separate
plot (Figure 5-3) i l ~ u s t r a ~ g filter performance in X and Y directions separately. The (.) curves indicate
botb the RMS error and the EE for the fUsed estimate in the specified direction. At first giance, it appears
that two separate pIots have been overlain on each axis; however, this is not the case. Because of the
common update schedule, two a posteriori estimates are produced at every sample tirne- The higher of
the two values is calculateci upon receipt of the range observation, which is pmcessed first by defauit.
The estimate is then impmved by pmcessing the imager angle observations 'immediately' thereafter-
- RMS Emr (Fused) 30 - -
Figure 5-2 Sequential Fiiter Performance (Total Position) Case 1 : 50 Runs, Radar T = 0.1 Cs], Imager T = 0.1 [s]
SequentiJ Fnter Performance in Y direction ( 50 Runs, raâT = 0.1 [sl, ImT = 0.1 Es] ) 25
Sequential Filter Performance in X direaion ( 50 Runs. r a d l = 0-1 Isl )
. ExpeaedEnor
*- RMS Emr i , - .
O O 1 2 3 4 5 6 7 8 9 1 O
l ime [s]
Figure 5-3 Sequential Filter Performance (Y and X Position) Case 1 : 50 Ruas. Radar T = 0.1 [SI, Imager T = 0.1 [s]
Processing the imager angles significantly improves the estimate in the early stages of the Y profile. while
the effect in the X direction is less pmnouaced. This phenornenon relates to the course's profile in
spherical coordinates; specificaily, beariag is the predominant factor in the Y direction. whiie range
predominates in the X direction. Thus, the hision offers a potential improvement in estimate accuracy that
is directly related to the orientation of the sensors witb rrspct to the target
Case 2: Decreasing the radar's update rate has a significant impact on filter performance (Figure 54).
Because a range observation is only available every ten simples, the total RMS position emr increases
periodically between radar updates, despite the imager's angular qom. in the extreme case . (at about
2 [s]), the estimates are outpedomed by the radar observations- While the RMSE eventuaily faiis below
15 [ml, the convergence is slow. Thus, although the sequential architecture is suited to the asynchronous
timing problem in theory, in practice it only perfoms well if the dimensionaiiy sufficient observations are
provided at a relatively hi& rate- With this point in mind, the performance of Candidates DF-1 and DF-3
are now discussed.
O O 1 2 3 4 5 6 7 8 9 1 O
Tirne [s]
Figure 5-4 Sequential Filter Performance (Total Position) Case 2: 50 Ruris, RadarT = 1.0 Cs], imager T = 0.1 Cs]
5.3 Candidate DF-1 : Static Decentralized Fusion.
Candidate DF- 1 corresponds to the filter developed in Section 3.2.1. It is the least computationally
demanding of the t h e architectures under consideration; however, it assumes that the target dynamics
will be linear in the image plane, whic h is not necessarily tnre. Only Case 1 is simuiated-
Case 1: The first point of interpst is the performance of the image filter (Figure 5-5). The 'y' axes of
the graphs are! lefi in 'pixel' coordinates, but the conversion to bcsring and eievation h m pixels may be
accomplished via Equatioa 4-2. hitially, the filter converges quickly. However, at the 3 [s] mark, the i
estimate diverges, foUowed by the j esthate at about 7 [s]. These points correspond mughly to the time
at which the target is crossing the imager's optical a i s , resulthg in a large bearing rate and non-Linear
image plane dyuamics. At this point the gain bas also dropped and the estimates cannot be reacquW
Even though the overail fused performance (Figure 5-6) is saIl quite reaso~ble. Candidate DF-1 is
eliminated fiom further consideration based on its unstable performance. in the last graph. the radar's
RMSE and EE are plotied to provide a baseline for judging the effectiveness of the fusion. Note thaî SF-
1 (Case 1) provided ody marpinalIy better results.
Imager Eladon Esümaüon ( 25 Rum. T4.l [SI ) 15, 4
!
Figure 5-5 Static Decentralized Filtet Performance (i and j Image Position) Case 1: 50 Runs, RadarT = 0.1 [s], Imager T = 0.1 [s]
Figure 5-6 Static Decentrdized Filter Pefiormance (Total Position) Case 1: 50 Rus, RadarT=0.1 [s], ImagerT = 0.1 [s]
5.4 Candidate DF-3: Dynamic Decei,tmlized Fusion
Decentraiized fbsion Candidate DF-3 was simuiated for both Cases 1 and 2.
Case 1: In Figure 5-7, DF-3's fused position performance ( - - ) is compared to the radar's EE (-) and
overall (fused) RMSE ( - ). The filter reaches a lower RMSE than SF-1 (about 3.5 Cm] ). This effect
can be attributed the fact h t the imaging sensor employs an aposteriori' range estimate in its input (i.e..
to fonn a dimensionally suficient rneasurement). Candidate DF-3 is now tested in Case 2.
Decentralized Fiiter Perfomance in Position ( 25 Runs. radar T = 0.1 [ç]. imager T = 0.1 [s] )
O 2 4 6 8 1 O Tirne [s]
I I I r
Figure 5-7 Dynamic Decentralized Filter Performance (Total Position) Case 1 : 25 Runs, Radar T = 0.1 [s] , Imager T = 0.1 [s]
-- Measurement Error (Radar) - - - - - - --- - --- - - - E>cpected Error (Radar)
- - - Wected Error (Imager) RMS Error [Fused)
Case 2: A decision for DF-3 over SF-1 c m o t be made on the bais of their results in the Case 1
scenax-io, due to the fact that in practice the radar will not provide updates at 0.1 Cs]. In fact, assuming
that the system operates in Surveillance mode, a simcant period of t h e could elapse between reports (
>l second ). Hence, judging on the basis of Case 2 is more appropriate. Under realistic conditions
Candidate DF-3 performed better than SF-1 (Figure 5-8). Although the fûsd RMSE decays to the sarne
value as in the sequential case, the decentralized filter outperforms a radar tracker ( - - ) by a
considerable margin at al1 times. Thus, based on its performance in Case 1 and Case 2 scenarios, the
decentralized dynamic filter is the best of the three architectures.
-
\ - '-4 3
* \
. \! -
?, \\ 4
'\' ,
-
5z --?&?Y =: - - __;- . - _ -- - .- - - ---
Figure 5-8 Dynamic Decentralized Filter Performance (Total Position) Case 2: 25 Runs, Radar T = 1.0 [s], Imager T = 0.1 [s]
A slight modification to the existing DF-3 algorithm will result in even better performance. In the previous
case (Figure 5-8), only the latest a posteriori range estimate is used at every imager sample instant.
without accounting for the motion of îhe target between radar scaas. However, if the a prion state h m
the radar is propagated iricrementally at the imager's sample rate, then the imager can provide more
accurate updates between radar detections (Figure 5-9). Successful employment of the modified
technique requires accurate image filter hitialization, and because the W s a prion covariance is
initially hi& the imager filter's covariance must k is increased at the begmning of the hision process to
prevent early divergence. This ability to accourit for the motion of the target distinguishes the
decenhalized architecture from the sequentiai architecture in a fiindamental way. in the latter case,
although the new angular measurements are available whenever the FLIR reports, the filter is unable to
propagate the target range before the aext radar detection, and consequently c m not profit from
knowledge of the target dynamics to the same extent-
Decentralized Fiîter Performance ( Radar T = 1 [s] Imager T = 0.1 [s] )
Figure 5-9 Dynamic Filter with Range Propagation (Totai Position) Case 2: 25 Runs, Radar T = 1 .O [s], imager T = 0.1 Cs]
80 - 1 1 1 1
Y---.,
On the basis of the preceding discussion. Architecture DF-3 (with propagation of radar range esthate) is
selected for use in the A\LRSS. Simulation has shown the dynamic decentralized technique to be more
accurate than the sequential füter in both ideal and practical timing senarios. Funherrnore. the technique
is inherently more robust as it incorporates a pair of fdters; thus, if the imager provides degraded
measurements, the radar tracks will remain unaffected.
70 t- i "-
60 n
€ u L
50 W c O :s 40 a Q)
g < 20
10
O
/ . - I - ..
, 4
t .- * .\ -
- l 1. - -. 1 -. .
/ -*. . -- - -- - -
I j
3 3 o r - T - - --- - - - _ _ - - - - - - - ,-;----- --- - *- - - - - -- ;y :>-Y - - -c _ -- - - - - - - -
t 1 I t L
O 2 4 6 8 10 Time [s]
Chapter 6 Validation with Real Data
In Chapter 5. the proposed algorihm were tested via Monte Car10 simulation, resulting in a choice for
Candidate DF-3 . However, the ultimate success of a prototype is determined by its peflormance under
actual operating conditions. For this reason, live trials were staged to collect realistic radar and infiared
surveillance data. Validation consists oÇ(i) processing the raw FLIR irnagery to identify the target
centroid fiom scan to scan and (ii) combining this idonnation with capnued radar data in o d e r to
measure the performance of the üacker over a single run. While ninning the test scenarïo was identified
as a secondary goal (outside the scope of this thesis), g d prelhhary d t s have been achieved with
regards to the image pmcessing. Tbese results are presented now as a confimation of the techniques
discussed in Chapter 4. The second stage of the validation process (see (ii)) is not yet cornplete;
however, it is still appropriate to discuss the testing procedure in terms of the performance measures to be
used. The frnal results will follow later in the year as a technical report released by the author to Defence
Research Establishment Valcartier (DREV).
6.1 Measures of Performance
Various Measures o f Performance (MOP) are available to assesses the effectiveness of a tracking
system. In the present case, we are interested in MOPs that describe behavioral functions [36].
Specifically, we wish to detennine how well the proposed system tracks a target in position and, when
more than one target is present, how well it handles the &ta association problern.
Estirnated position accuracy can be detennined in two ways: by comparing the estimated tmck to the
ground tmth or by forming the NIS (Section 2.1.1). The NIS is usefiil as it relates the measurement
residuals to the total expected e m r covariance, establishing for a desired confidence Level whether or not
a the filter is behaving consistently. The NIS is a coarse test because it assumes the rnie state is
unknown. Given the ground truth, the Nonnalized Estimation E m r Squared (NEES) can be used as a
more precise test- The NEES compares the difference between hue and estimated States by nonnalizing
the estimation error z ( k ) by the estimate e m r covariance over N, nias. Fonnally, the NEES is detined
w here
There is an obvious similarity between the NEES and the MSE used in Monte Carlo simulations.
Because the NEES is chi square distributed, it can be used not only to calculate the estimation error, but
also to determine whether the e m r is consistent with the system. Both the NIS and the NEES are
appropriate MOPs for the validation pmcess-
6.2 Setup and Procedure
The triais, conducted in a nual cornmunity near Ottawa, Oat., consist of singie and multi-target scenarios-
Both were run on a pre-set route (Figure 6-1). The target(s) included a Dodge Dakota and a
Volkswagon Golf, whose radiation profiles are similar to the Forces' LSVW (truck) and Iltis (jeep). The
targets traveled at constant çpeeds between pre-marked 'way points' (WP) along a relatively straight side-
road. They were observed by a LAV Recce positioned roughly a kilometer away. The sensors' Iines of
sight (LOS) were CO-aligne& and theu arcs chosen such that the radar's search volume enveloped the
imager's. The ground m t h was established by recording the grid of each WP using a GPS with an e m r
radius of 10 meters. This information was used initially to align the radar and inffaced sensors. It is also
used by the NEES. The following section outlines the procedure used to capture the radar and infrared
data.
6.3 Infrared Data Proœssing
The infrared imagery was originally recorded in Hi-8 video format. Later, the d o g u e data was
sampled at 15 frames per second and written to compact disk as a series of 640 x 480 pixel bitmaps. The
bitmaps were impoited into MatlabTM for segmentation, quantization and clustering. The proposed
AlLRSS would include a digital image processing facility and would perfom these operations on-line.
During the test, the CRT monitor contrast and brightness were set visualiy to the highest SNR without
distortion. Because the theoretical target and background radiances were unknown, the required
probability distribution fùnctions (for target and background) were be obtained offline by sampling regions
of interest within a test image (Figure 6-2).
Figure 6-1
Figure 6-2
INITIAL INFFMRED IMAGE
256 Level Bitrnap
i&ared Video Image Captwed by LAV Recce
-64-
in Figure 6-2 (captured dwing a trial run) two targets are heading towards each other in the vicinity of the
cross hatch. Apart h m the target statistics, it would be usefil to establish the distribution of the
background in t ems of the sky (top horizontal band), the tree line (middle band) and the foreground
(lowest band). A histogram (Figure 6-3) was constnicted by sarnpling the image in the appropriate area
with a rectaogular window. The target has a sample mean of 104 units and sample variance of 5 uni&
squared. To extract the target location it was necessary to generate a set of thresholds for a given PT and
P,. These quantities were determined by integrating the regions beneath the appropriate target and
background probability distributions for various IL and 1, (In this case the distributions were treated as
Gaussian).
INTENSITY DISTRIBUTION Sky. Treeline. Ground
45 ( I 1 1 1 1 I 1 1
Figure 6-3
TREES mean: 82
GROUND var:7 mean: 70
SKY maan: var: 3
70 80 90 100 110 120 130 Gray Scale Intensity
Histogram of Typical Pixel Intensities
Figure 6 4 illustrates the effect of an overly wide passband 1, E (80-120), resulting in the deteciion of a
large portion of the me-line and sky. AAer tightening the bandpass to Iu ~(92-115) - corresponding to PT
> 0.99, P, < 0.05 - the target centroid was identified more clearly; however, a sparse distribution of
background pixels remained (Figure 6-5).
50
1 O0
150
200
250
300
350
400
450
Figure 6-4
AFTER BANDPASS IH=80 IH=120
Intensity Bandpass with Overly Wide Threshold
AFTER BANDPASS lH=92 l,,=115
100 200 300 400 500 600
Figure 6-5 Intensity Bandpass for PT > 0.95 0.05
To eliminate detections that were W e l y to originate h m the target, the remaining pixels in the search
area were clustered with a cluster distance of 2.8 pixels (Figure 6-6)- As expected, the largest cluster
corresponded to the target vehicle (containitg 13 pixels).
(300. 325)
Figure 4-6
CLUSTERS IN SEARCH WINDOW (200. 425)
Clustered Pixels
Eiiminating al1 clusters with a membership of less than 8 pixels multed in a complete extraction of the
target centroid (Figure 6-7).
ISOLATED CENTROID (pi = 424. pj = 368) $ùels] <=> (theta = 15, phi = 6) [ mrad]
100 200 300 400 500 600 j [pixels]
Figure 6-7 [solated Centroid
6.4 Radar Data
Recording the radar information for each scenario represented a significant hurdle because, like the FLIR
data, it is analog. A two pronged approach was devised to produce the required strings of discrete time-
tagged measurements: First the target's changing location was recorded by an on-board facility. which
cannot be described in any detail. Although in Swveillance mode this information is accurate to within
roughly one tenth of a beamwidth, it lacks temporal significance. To detexmine the tirne of each update, it
was assurned that each detection was made when the antenna's LOS coincided with the target's bearing.
This allowed the temporal association to be made by simply recotding the antenna's orientation with
respect to time - accomplished by fixing a variable resistor to the rotating console, and connecting it in
series to a dc voltage source (Figure 6-8). A strip chart recorder was then c o ~ e c t e d across the
potentiometer to measure the change in voltage as the radar swung through its arcs. Because the anteana
rotates at a fixed rate, the relationship between bearing, f3 , and voltage, V , is linear. Thus, the required
relationship can be detemineci h m the dope (dependence on tirne ornittecl):
Figure 6-8 Radar Senip
Chapter 7 Summary
7.1 Objectives Met
The objective of this thesis was to devdop an eficient, r o b t and Mective targer tracking
architecture that corn bines active in formation fiom a Moving Target Indicator m) radar and
passive infomation&m a Fonvard Looking Infiared (FUR) imager.
Meeting the primary objective entailed (i) evaluating various h i o u architectures; (ii) selecting an
appropriate tracking fiIter, (iiï) choosing maneuver detectiod compensation and track initiation schemes
(iv) developing a method to extract the target's angular position h m analog infiared images; and (v)
evaluating the performance of the candidate algorithms via Monte Carlo simulation.
7.2 Results and Design Choices
Several candidate architectures were initially considered. Of these, three were developed as viable
approaches: A Sequeutid Filter (SF-1) showed initial promise because of its ability to acco1111110date
dimensionally insufficierit measurernent models and process observations as they arrive. A static
decentralized architecture (DF-1) was also considered which employed a Linear image plane filter to
process the optical data. Although it was efficient, the technique was ultimately flawed because a
constant velocity profile in Cartesian Coordinates will not be viewed as a Iinear fimction by the FLlR.
FinalIy, a dynamic approach to decentralized h i o a was developed @F-3) which augments the
insufficient imager measurements with the latest range estimate thus forming a complete point that can be
processed locally and then fùsed at the global node. Al1 three candidates were simuiated, and on the basis
of its superior performance, the third option @F-3) was selected for use in the 'Advanced' LAV Recce
Surveillance Suite (ALRSS). This architecture is inherently more robust than the single filter approach,
and can be adapted advantageousty to realistic timing scenarios.
The Modified Baheti filter was selected as the most appropriate fiIter, which takes advantage of the linear
Cartesian state dynamics to produce significant computational swings. Maneuver compensation was
treated bnefly, and the straight forward strategy of reinitializing the local filters' covariance matrices was
deemed appropnate for this application. Finally, a sequential track initiation scheme was outlined to
complete the initial design.
This thesis is f m and foremost a practical application. In this regard, a method was required to extract
kinematic information h m aoisy FLIR imagery pnor to the fhsion process. A mapping between the
image screen and Spherical Coordinates was presented, which assumes the target centroid bas been
isolated as a mean pixel position with assoçiated variance. The isolation process involves segmentation,
quantization and clustering, as demonstrated on a typical infiared image. These techniques were
success full y demonstrated on actual infiared surveiliance data.
This thesis is limited to a simulation of the single target case; however, an extension to the multiple target
was also discussed. Nearest Neighbour Data Association is favored for the assignment process. Finally,
no design is complete without a field test. To judge the effectiveness of the proposed architecture, the
LAV Recce was tested in a realistic surveillance exercise that will eventually yield a complete set of pre-
processed data with which to validate this and future designs.
7.3 Novel Features of this Thesis
Military Application. The LAV Recce currentiy has m target tracking capability, w hatsoever. This
thesis not only presents a radar tracker, it also incorporates the FLIR to increase the accuracy of the
estimation process.
Fusion Architecture. The tracking system is based on a decentralized &ion architecture, whereas the
de facto approach to combine radar and infrared information is sequential fusion. The decentralized
approach proposed in this thesis is novel in that a common state mode1 cannot be assumed. Specifically, it
is necessary to augment the KIR updates with a prion range estimates to fonn a sufficient point pnor to
sensor-level filtering and global-level fusion. To the author's best Imowledge, this strategy bas not been
suggested in the literature for the current application.
Angle Mapping. Before proceeding with the fusion, locally measured aogular quantities need to be
mapped between coordinate frames. The transfonn is straight forward in the polar case; however, when
the local and global points are located in a sphencal volume, the relationship is not straightfonuard. Using
a geometric rotation mapping typically seen in mbotics[37], an exact transfomi [3 11 is developed (Annex
B) that improves upon a published method that is proven to be at best appcoximate [24].
7.4 Recommendation for Continued Invrrtigiaon
This thesis has attempted to elucidate the enonnous potential of the LAV Recce's Surveillance Suite,
Several other opportunities exist to empioy sensor &ta fbsion to great advantage. Adding the day camera
to the tracking loop, for instance would provide a means of identifjing targets and detecting maneuvers,
Otherwise, a pixel Ievel h i o n between this seasor and the FLR would greatly increase the operator's
ability to perfonn surveiLlance under marginal lighting and atmospheric conditions- However, perhaps the
most challenging extension involves a fiil1 assault on the practical aspects of the &îa association problem.
While the multitarget case has been discussed by many authors, attacked h m various angles, it has yet to
be championed in a simple, robust and effective manner,
Annex A Project Background
Location
LFTSC
RMC
Meaford
DREV
Ottawa
Hosu Eveat
LCol Lord
LCol
Carruthers
Maj Muir
LAV Recce
Tactical
Evaluation
J- Cniickshank
Y. de Villers
LCol
Carruthers
Collected initial documentation on LAV
Race Surveillance Suite (LRSS).
Discussed current DREV mandate to make
immediate improvements to LRSS.
Presented ideas to PM0 UV,
First hand inspection/ demonstration of LRSS
Collected sensor specifications.
Attended LRSS field test.
Solicited feedback on proposeci ideas h m
LRSS operators.
Discussed data fonnat with J, Cmickshank.
- - - --
a Presented current proposal to scientific
authorities in LAV Recce research group.
Discussed hardware requirements.
Collected short sequence of sample image
data
Arranged for more comprehensive copy of
FLR &ta to be recorded on CD for processing
at RMC-
Collection of real data (involves nrnniag a
preplanned scenario and recording data with
LAV Recce sensors-)
Annex B An Alternate Technique for Transferring Angular Measurements Between Coordinate Frames [2 11
The augmenred rotation mapping can be used whenever angular information needs to be mapped
from one coordinate fnune to another (Figure B-1). Define the local and global coordinate frames
according to Figure B-1. To map fiom the local to the global frame, follow the steps in Table B-1.
h = ronge dtoget mtroid & = M g (LOS) 8c = bearing (œntroid) ips = elmation (LOS)
A = local quantity w.r.t. LOS
Figure B-1 Relationship Between Local and Inenial Angles [24]
- -
Step
1
Description Notes
Measure target location {Agc. AqJ E 3t2.
Assign r, = 1 to form {r,, Agc , AQJ E 3t3.
Apply spherical-Cartesian transfonnation to
rom *P = f2px, *P,,, 2pzl
r, is arbitrary and does not affect
W C . A d -
Apply rotation rnap~ing If origins of Frames (1) and (2) are
not collocated, rotation mapping
1 R is replaced wit h
transfonnation mapping [37l
a) Convert ' P = { 'p, . 'p, . 'pz] to sphedcal
coordinates { r,, 8 . cp)-
rc is independent of (8 . Q}.
b) Disregard rc, and extract inertial angles (8 , Q).
Table B-1 Augmented Rotation Mapping [3 11
The augmenteci rotation mapphg bas been pmposed by the author [21] as an exact method to
accomplish the required cmrdinate fhme transformation. It is a suitable replacement for an altemate
technique that has appeared in the recent literahw, which is shown to be at best approximate.
Specifically, it is stated in [ ] that the angular location of a target with respect to an arbitrary inertial
coordinate fiame (8 , cp ) , is equal to the sensor's line-of-sight, (O,, cp,), plus the angular location of the
target (centroid) with respect to the sensor's coordinate fiame, {AOC . Acp, 1; Le., that
O= es+ A 8, @-la)
and
v= q+ A 9, (B-1 b)
where 0 denotes bearing and cp denotes elevatioo. In [ 1, Equations (B-La) and (E3-16) are employed
without restriction; however, the technique does not hold unless <p, = O , and /or Ag, = 0,
Proof
Assume Equations (B-la) and (B-1 b) can be utilized without restriction to transfocm angular quantities
fiom a local coordinate frame, Frame (2 ) , to an inertial coordinate frame, Frame { 1 ) (see Figure B- 1).
The assignment of an arbitrary range, rc, to the inertial mgles, (0 , cp) , establishes a complete point which
can be expressed in Cartesian coordinates. For convenience. let rc = 1, and denote the inertial point by
'P ' , where
Now, it should be possible to generate an equivalent inertial point, say 'P.
by transforming a locally measured point. %
with the rotation mapping presented in [ 1; Le.,
As the local sensor oaly measures angles, 2P E 93' must be fomed by assigning an arbitmy m g e to
(A@, , Acp, ) , as in the previous constniction of ' P r b m {O , cp ). Since axis x, (the x-axis of Frame
(2)) corresponds to the sensor's Lue-oGsight, ;R represents a rotation of the inertial !tame about y, by
-cp, and then about 2, by 8,. By the X-Y-Z fixed angle convention 1281,
Thus, provided Equations @la) and (B-lb) hold without restriction, one would expect
'p., lp W)
However, a close examination of Equation (B-4) reveals that the equality does not hold true in general.
Expanding the x, y, and z components of 'P* and 'P with the aid of standard trigonometric identities yields
a set of three equalities. Starting with the z components. it is cequired that ' p ;= ' p z . where
1 P. r = COS(Q? a. sin(^, )FOS(A <p., )
Comparing Equations (B-sa) and (B-5b) reveals immediately that
' P * ' P
except when AOC = O and/ or cp, = O. Equating the rernaining x and y ternis in the position vectors
confinns this constraint. i
References
Y. de Villers, private conversation with D. van Huyssteen and M. Farooq, Oct., 1997.
A.M Marston, R.A. Schell and J.E. Webb, "Cperational Concept Document Coyote Information Management System," Department of Applied Military Science, Royal Military College of Canada, 3000-7 (LJTSC-l), May 1996.
J. Gagliarducci, "Interface Control Specification for the Radar Extemai Interface," Electronics and Space Corp., St. Louis, Mo., 18 May 1994.
"Equipment Description Light Annoured Vebicle (UV) Reconnaissance (Recce) Mast Mounted Surveiilance System (MMSS) and Remote Mounted Surveillance System (RMSS), C-59-867- 000/MD-00 1" Canadian Department of National Defence, Project Management Office LAV, Jan 1997.
J, Llinas and E- Waltz, Multisensor Data Fmion. Boston, MA: Artech House, 1990.
D. Hall and J. Liinas, "An Introduction to Multisensor Data Fusion," Proceedings of the IEEE, vol. 85, no.1, Jan. 1997, p.7.
Y. Bar-Shalom and X. Li, Multitarget-Multisensor Tracking: Principles and Techniques. 1995, Y .B.S. Publishing, Storrs, CT, 1995.
S.W. Sorenson, "Least-squares estimation: from Gauss to Kaiman," IEEE Specîmm, vol. 7, pp. 63-68, July, 1970.
Applied Optimal Estimation, A. Gelb, ed., M.I.T. Press. Cambridge, Massachusetts, 1974.
A.H. Jazwinski, Stochastic P rocesses and Filtering Theoty, Academic Press, New York, 1970.
J.H. Taylor, "The Cramer-Rao Estimation Error Lower Bound Computation for Deterministic Nonlinear Systems," IEEE Transactions on Automatic Control, Vol. AC-24, No. 2, April 1979.
M.R. Spiegal, Schaunt 's Outline Series: Mathematical Handbook of Formulas and Tables, McGraw-Hill Book Company, New York, 1968.
M. Farooq, A. Rouhi and S.S. Lim, "Analysis of suboptimal filtering algorithms for target tracking," Elsevier Signal Processing, vol. 30,110-2, Jan. 1993.
Y. Bar-shalom, ICC- Chang, "Tracking a Maneuvering Target Using Input Estimation Versus the Inte rac ting Multiple Mode1 Algorithm," IEEE Transactions on Aerospace and Eleclronic Systems, Vol. 2, AES-25 No.2, March, 1989.
Y. Bar-Shalom and K- Binniwaf, "Variable Dimension Filter for Maneuvering Target Tracking," IEEE Transactions on Aerospace and Electronic System, Vol. 2, AES-18 NOS, September, 1982.
S. Bnider, S. Lim, B. Johnson and M. Farooq? "A Study of Multiple Target Tracking in a Cluttered Environment," E.E. Tech Report No. 921 4, Royal Military CoUege of Canada, Kingston, Ont., August, 1992.
C. Chong., et. al., "Distributed Multitarget Multisensor Tracking," in Multitarget-Multisensor Tracking: Advanced Applications, Y- Bar Shalom, cd, Artech House, Boston, 1990.
S.S. Blackrnan and T.J. Broida, "Multiple Sensor Data Fusion in Aerospace Applications," Journal of Robotic Systemr, vol. 7(3), pp. 445-485, 1 WO.
Robert J. Fitzgerald, "Development of Practical PDA Logic for Multitarget Tracking by Microprocessor," in Multitarget-Multiseosot Tracking: Advanced Applications, Y. Bar Shalom, ed., Artech House, Boston, 1990
S. Bnider, M. Farooq, M. Bayoumi, "Robotic Heterogenous Mdti-Sensor Fusion with Spacial and Temporal Alignment," Proceeditigs of the 3(Y"onference on Decision and Control, Dec., 199 1.
S. Bruder and M. Farooq, "Sensor Fusion Techniques and a Hybnd Multi-Sensor Multi-Target Tracker," EE Tech Report no. 90/3, Royal Military College of Canada, Kingston, Ont., Aug. 1992.
H. Hashemipour, S. Roy, and A, Laub, "Decentralized Structures for Parallel Kalman Filtering," IEEE Transactions on Automatic Control, vol. 3 3, No 1, Jan., 1 988.
M. Krieg and D. Gray, "Radar and Optical Tmck Fusion Using Real Data," in Proc First Annual Australian Data Fusion Symposium, Adelaide, South Australia, 1997.
J. Romine and E. Kamen, ''Modelting and Fusion of Radar and Imaging Sensor Data for Target Tracking," Optical Engineering? vol. 35, no. 3, pp. 659-672, Mar., 1996.
E. Oron, A. Kumar, and Y- Bar-Shalom, "Precision Tracking with Segmentation for Imaging Sensors," IEEE Trans. on Aerospace and Efectronic System, July, 1 993, pp. 977-985.
V. Aidala and S. Hammel, "Utilization of Modified Polar Coordinates for Bearings-Only Tracking," IEEE Trans. Automatic Control., Vol., AC-28, No.3, pp. 283-294, 1983.
Burgeois, F. and J-C. LaSalle, " An Extension of the Munkres Algorithm for the Assignment Problem to RectanguIar Matrices," Communications of the ACM, vol. 14, Dec. 197 1, pp. 802-806.
C.B Chang and L.C. Youens, "Measurement Tmckuig for Multiple Sensor Tracking in a Dense Target Environmenf" IEEE Tram- Automatic Conaof, vol. AC-27, December. 1982, pp. 1250- 1252.
Y. Bar-Shalom and T.E. Fortnian, Tracking and Data Association, Academic Press, San Diego, 1988.
B.E.A Saleh and M.C. Teich, Fundamentds of Photonics, John Wiley and Sono, Toronto, 199 1.
[3 11 D. van Huyssteen and M. Famoq, "An altemate technique for transfemng angles between coordinate *es," Opt. Eng. (pending review).
[32] Y. Bar S halom, Multiiensor-MuItitarget Tracking, Advanced Applications, Short Course Notes, UCLA.
[33] R.D. Hudson, I ' a r e d System Engineering, John Wiley and Sons, Toronto, 1969,
[34] A. Kumar, et. ai., "Precision Tracking Based on Segmentation with Optimal Layering for ïmaging Sensors," SPIE Vol, 1954, pp. 244-255.
[3 51 M.R. Anderberg, ed., Cluster Analysis for Applications. Academic Press, 1973.
1361 D-L- Hall, Mathematical Techniques in Muftisensor Data Fusion, Artech House Inc., Boston. I W i .
[37] 1.1. Craig, Introduction to Robotics. Addison-Wesley Publishing Company, New York, 1989, Chap. 2,