real world

data fusion

Fred Daum

15 June 2012

data fusion

Copyright © 2012 Raytheon Company. All rights reserved.

Customer Success Is Our Mission is a trademark of Raytheon Company.

1

PATRIOT

Firefinder

F/A-18

Global HawkFirefinderPAVE PAWS

Global Hawk

2

theoretical optimal multi-sensor data fusion

fusion of measurements

performance

fusion of

tracks

interesting parameter3

real world multi-sensor data fusion

fusion of tracks

performance

fusion of measurements

interesting parameter4

real world issues in multi-sensor data fusion

• limited resolution of sensor data

• residual bias & drift errors of sensor data

• physics of real world bias & drift errors

• data association errors

• not all objects detected & resolved & reported by sensor • not all objects detected & resolved & reported by sensor

A are detected & resolved & reported by sensor B

• error covariance matrix inconsistency

• nonlinear & non-Gaussian filtering errors

• ill-conditioning

• limited bandwidth & latency & poor connectivity & poor

content of data communication between multiple sensors

• other5

Séminaire de Statistique. Le 19/06/2012 à 11h00,

Salle de séminaire du 1er étage, bât.1R3

Frederic Daum : Nonlinear filters with particle flow

We have invented a new particle filter, which improves accuracy by several orders of magnitude

compared with the extended Kalman filter for difficult nonlinear problems. Our filter runs many

orders of magnitude faster than standard particle filters for problems with dimension higher than

6

orders of magnitude faster than standard particle filters for problems with dimension higher than

four. We do not resample particles, and we do not use any proposal density, which is a radical

departure from other particle filters. We show very interesting movies of particle flow and many

numerical results. The key idea is to compute Bayes’ rule using a flow of particles rather than as a

point wise multiplication; this solves the well known problem of “particle degeneracy”. Our

derivation is based on freshman calculus and physics. This talk is for normal engineers who do not

have log-homotopy for breakfast.

multi-sensor data fusion literature

publication sensor bias sensor resolution

physics of real world bias

Fusion Conferences 1% 0% 0%

Handbook of Multisensor Data Fusion

0% 0% 0%

Fusion

Sam Blackman’s books

1% 1% 0%

Yaakov Bar-Shalom’sbooks

2% 3% 0%

Oliver Drummond’s SPIE conferences

1% 1% 0%

IEEE Aerospace & Systems Transactions

1% 1% 0%

7

bias can ruin multi-sensor fusion

300 total targets: 30 missiles, 10 targets per missile

Position error σ = 100m, Separation of targets in missile complex = 500m, 1500m

60

70

80

90

100

% C

orr

ec

t a

ss

ign

me

nts

GNPL

0

10

20

30

40

50

60

1 10 100 1000 10000

Magnitude of bias (m)

% C

orr

ec

t a

ss

ign

me

nts

JVC

GNPL jointly

optimizes the

bias estimates &

data association

8

GNPL vs. JVC with Bias

• 7 remote tracks and 29 local tracks (2 of the remote tracks have no

local track) with residual radar bias

10

GNPLJVC

2 4 6 8 10 12

2

4

6

8

2 4 6 8 10 12

Perfect

fusion

All fusion is

incorrect

9

GNPL explicitly models bias*

• Score function jointly optimizes data association and bias

estimation:

( )[ ] [ ]∑

−−

=

≠+−−−=

+=

−−=

miii

T

iMT

a

iaii

iaii

iag

iaSxSxRxRxJ

QPS

xBAx

1

1

)(

)(

0)(

0)(ln2ln

δδπ

δestimated bias

• The optimal estimate of bias is:

[ ] ∑=

=i

aiag1 0)(

[ ] [ ] [ ]

∑

=

≠−+

∑ ++=

=

−−

=

−−m

1i

ai1

ai

1m

1i

1ai

1

0)i(a0

0)i(aBAQPQPRx ii

i

10*Mark Levedahl, proceedings of SPIE conference 2002.

should be computed

adaptively

comparison of algorithms

algorithm bias estimation? performance

1. GNPL + adaptive gating yes (jointly with

association)

2. iterative bias estimation & JVC yes

3. histogram of bias over association hypotheses

yes

4. covariance inflation & JVC no

5. association of objects (JVC) no

11

GNPL with adaptive gating is much better than Iterative JVC

(d = 6 & N = 10 objects)

0 10 200

0.2

0.4

0.6

0.8

1

Perc

ent C

orr

ect M

atc

h

100% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

190% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

180% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

170% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

160% Track Overlap

Pro

babili

ty o

f P

CA

0 10 200

GNP L

IJV C

0 10 200

0 10 200

NNp / 1-sigma

0 10 200

0 10 200

0 10 200

0.2

0.4

0.6

0.8

1

Pe

rce

nt C

orr

ect M

atc

h

50% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

140% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

1

NNp / 1-sigma

30% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

120% Track Overlap

0 10 200

0.2

0.4

0.6

0.8

110% Track Overlap

GNP L

IJV C

Pro

babili

ty o

f P

CA

12

GNPL with adaptive gating is much better than iterative JVC

(d = 6 & N = 10 objects)

• Delta (GNPL – IJVC) 10 Object Association Performance

– 1σ Bias Error = 300m , 1σ Track Error = 100m, TOL = track overlap, Monte Carlo runs = 200

0.25

0.3

0.35

0.4

Pe

rce

nt C

orr

ect A

sso

cia

tion

Delta Percent Correct Association (GNPL minus IJVC)

100% TOL

90% TOL

80% TOL

70% TOL

60% TOL

Pro

babili

ty o

f P

CA

0 5 10 15 20 25-0.05

0

0.05

0.1

0.15

0.2

0.25

NNp / 1-sigma

Pe

rce

nt C

orr

ect A

sso

cia

tion

50% TOL

40% TOL

30% TOL

20% TOL

10% TOL

Pro

babili

ty o

f P

CA

13

physics of real world “bias”

physical source of error bias error varies with

1. misalignment of electrical boresite & mechanical IMU axes

azimuth & elevation and/or array sine space (u & v)

2. IMU drift time

3. monopulse slope error angle from beam center

4. scan dependent monopulse bias scan angles from array boresite, temperature of array face & refractivity near array face

5. tropospheric refraction azimuth & elevation & time

6. radome refraction & reflections azimuth & elevation and moisture on radome

14

antennanear ty refractivi and

antenna of re temperatuoffunction

:bias monopulsedependent scan

=

+=

+=

k

kvvv

kuuu

m

m

15

what is “resolution”?

Consider a sensor with resolution volume V, and two

objects A and B. We say that the sensor “resolves” A

and B if the resolution volumes are disjoint.

• A

• B

• A

• B

Resolved Unresolved

16

Probabilities of Resolution and Data Association for

One-Dimensional Measurements

17

simple back-of-the-envelope formulas

PR = probability of resolution

PR ≈ exp (-λV)

λ = density of objects

V = volume of sensor resolution cell

P = probability of correct data associationPDA = probability of correct data association

PDA ≈ exp (-cN)

N = average number of objects in the one-sigma prediction error volume

c = constant (on the order of unity) for a given dimension

18

fundamental theorem of multiple target tracking*

PR < PDA

P = probability of resolutionPR = probability of resolution

PDA = probability of correct data association

*assumes that prediction errors are dominated by noise rather than

target maneuvers. see Daum & Fitzgerald, “The importance of

resolution,” Proceedings of SPIE Conference on signal & data

processing, vol. 2235, pages 329-338, April 1994.19

example of data association problemR

an

ge

t t t t t t t t0 1 2 3 4 5 6 7

Time

Ran

ge

20

solution to data association problemR

an

ge

0 1 2 3 4 5 6 7

Time

Ran

ge

t t t t t t t t

21

example of resolution problem

Ran

ge

t t t t t t t t0 1 2 3 4 5 6 7

Time

Ran

ge

22

0.0

111.6

208.9

273.0

600

540

480

420

RE

FE

RE

NC

E T

RA

CK

AL

TIT

UD

E (

km

)

TIM

E (

s T

AL

O)

(43

80

0.7

7s

UT

C)

BRV

CSO CLOUD

PROPELLANT

DEBRIS

GCS

CSO CANISTERS

CSO PANCAKE

0.0

111.6

208.9

273.0

600

540

480

420

RE

FE

RE

NC

E T

RA

CK

AL

TIT

UD

E (

km

)

TIM

E (

s T

AL

O)

(43

80

0.7

7s

UT

C)

BRV

CSO CLOUD

PROPELLANT

DEBRIS

GCS

CSO CANISTERS

CSO PANCAKE

304.8

304.5

271.9

207.0

-1.6 0.0 1.6 3.2 4.8

360

300

240

180

120

RE

FE

RE

NC

E T

RA

CK

AL

TIT

UD

E (

km

)

RANGE RELATIVE TO REFERENCE TRACK (km)

TIM

E (

s T

AL

O)

(43

80

0.7

7s

UT

C)

BRVPROPELLANT

DEBRIS

M57

BRV / GCS

UNITARY

304.8

304.5

271.9

207.0

-1.6 0.0 1.6 3.2 4.8

360

300

240

180

120

RE

FE

RE

NC

E T

RA

CK

AL

TIT

UD

E (

km

)

RANGE RELATIVE TO REFERENCE TRACK (km)

TIM

E (

s T

AL

O)

(43

80

0.7

7s

UT

C)

BRVPROPELLANT

DEBRIS

M57

BRV / GCS

UNITARY

23

normal debris for ICBM

24

performance improvements due to explicitly

modeling unresolved data

without explicit model of unresolved measurements

with explicit model of unresolved measurements

Koch & van Keuk (1997) unstable tracks and bad estimation accuracy

stable tracks with excellent estimation accuracyaccuracy

Blom & Bloem (2005) 50 % probability of track coalescence

2 % probability of track coalescence

Daum (1986) 50 % probability of maintaining track and poor velocity estimation accuracy

99 % probability of maintaining track and excellent velocity estimation accuracy

25

question:

Suppose that there are 1,000 objects in a single radar

beam, randomly and uniformly distributed in range

over 100 km. Suppose that the radar range

resolution is 10 m.

What is the probability that a given object is

resolved from all other objects?resolved from all other objects?

(a) 0.98

(b) 0.82

(c) 0.26

(d) 0.0

26

answer*

λλλλ = 1,000 objects/100 km

λλλλ = 1 object/100 m

V = 2∆∆∆∆R

V = 20 mV = 20 m

PR ≈≈≈≈ exp (-λλλλV)

PR ≈≈≈≈ exp (-20 m/100 m)

PR ≈≈≈≈ exp (-0.2)

PR ≈≈≈≈ 0.82

*pessimistically assuming no Doppler resolution 27

resolved measurements are good & unresolved data

are essentially useless

Data

Measurement Accuracy of Radar

Data

Value for

Discrimination and Tracking

Resolved

Unbiased and limited by noise approximately as per standard

Good (assuming adequate signal-approximately as per standard

handbook formulas

adequate signal-to-noise ratio)

Unresolved

Strongly biased with large

variances, with errors roughly an order of magnitude larger than

resolved measurements

Essentially worthless

28

resolved measurements are good

Data Measurement Accuracy of Radar Data

Value for Discrimination and Tracking

Resolved

Unbiased and limited by noise approximately as per Cramér-Rao

bound:

NS2nT4

12

NS2B

2cRR

ππππ

λλλλ≈≈≈≈σσσσ≈≈≈≈σσσσ

Good (assuming adequate signal-

to-noise ratio) •

NS2

1

NS2

ANS2kNS2k

NS2nT4NS2B

A

m

3EL

m

3AZ

≈≈≈≈σσσσ≈≈≈≈σσσσ

θθθθ≈≈≈≈σσσσ

θθθθ≈≈≈≈σσσσ

ππππ

φφφφ

Unresolved

Strongly biased with large variances:

12

2badvery

1212B

2c

A

3ELAZR

ππππ≈≈≈≈σσσσ≈≈≈≈σσσσ

θθθθ≈≈≈≈σσσσ≈≈≈≈σσσσ≈≈≈≈σσσσ

φφφφ

Essentially worthless

29

definitions of symbols

λ = radar wavelength

c = speed of light

B = chirp bandwidth of waveform

S/N = signal-to-noise ratio

T = coherent integration time

L = width of radar antennaL = width of radar antenna

D = height of radar antenna

R = slant range

R = range rate

A = target amplitude

φ = target phase

n = number of pulses coherently integrated

DorL

2k

3

m

λλλλλλλλ====θθθθ

ππππ====

•

30

radar resolution formulas*

dimension

formula for Rayleigh

resolution

examples

azimuth LAZ λλλλ====∆∆∆∆ λλλλ = 3 cm and L = 10 m ⇒⇒⇒⇒ ∆∆∆∆AZ = 3 mrad

elevation DEL λλλλ====∆∆∆∆ λλλλ = 3 cm and D = 5 m ⇒⇒⇒⇒ ∆∆∆∆EL = 6 mrad elevation DEL λλλλ====∆∆∆∆ λλλλ = 3 cm and D = 5 m ⇒⇒⇒⇒ ∆∆∆∆EL = 6 mrad

range B2cR ====∆∆∆∆ B = 10 MHz ⇒⇒⇒⇒ ∆∆∆∆R = 15 m

B = 100 MHz ⇒⇒⇒⇒ ∆∆∆∆R = 1.5 m

B = 1000 MHz ⇒⇒⇒⇒ ∆∆∆∆R = 15 cm

Doppler T2R λλλλ====∆∆∆∆ secm3.0Rsecm100Tandcm3

secm3Rsecm10Tandcm3====∆∆∆∆⇒⇒⇒⇒========λλλλ

====∆∆∆∆⇒⇒⇒⇒========λλλλ

• •

•

*Assuming no weighting and no super resolution

31

industrial strength data fusion

explicit model

of resolution

explicit model

of bias in data

association

use sensor A data

to diagnoseunresolved

data forsensor B

robust

fusionalgorithm

monopulsequadrature

for unresolved

data

use chi-square

residuals for unresolved

data

retrodictionof

unresolved data

predictionof

unresolved data

32

• Wolfgang Koch & Günter van Keuk, “Multiple hypothesis track maintenance

with possibly unresolved measurements,” IEEE Trans. AES, July 1997.

• Shozo Mori & Chee Chong, “Multiple Target Tracking with Possibly Merged

Measurements,” SPIE Conference, San Diego, August 2005.

• Henk Blom & Edwin Bloem, “Descriptor system approach towards multi-

target tracking under limited sensor resolution,” March 2005.

• Mark Levedahl, “An explicit pattern matching assignment algorithm,”

Proceedings of SPIE Conference on Signal and Data Processing, Volume

4728, 2002.

• Martin Dana, “Registration: A prerequisite for data fusion,” Chapter 5 in

Multi-target Multi-sensor Tracking, Volume I, edited by Yaakov Bar-Shalom,

Artech House Inc., 1990.

• Fred Daum, “A System Approach to Multi-target Tracking,” Chapter 6 in

Multi-target Multi-sensor Tracking, Volume II, edited by Yaakov Bar-Shalom,

Artech House, Inc., 1992.

• Dimitri Papageogiou & John Sergi, “Simultaneous track-to-track association

and bias removal,” Proceedings of IEEE Aerospace Conference, Big Ski

Montana, March 2008.33

Séminaire de Statistique. Le 19/06/2012 à 11h00,

Salle de séminaire du 1er étage, bât.1R3

Frederic Daum : Nonlinear filters with particle flow

We have invented a new particle filter, which improves accuracy by several orders of magnitude

compared with the extended Kalman filter for difficult nonlinear problems. Our filter runs many

orders of magnitude faster than standard particle filters for problems with dimension higher than

34

orders of magnitude faster than standard particle filters for problems with dimension higher than

four. We do not resample particles, and we do not use any proposal density, which is a radical

departure from other particle filters. We show very interesting movies of particle flow and many

numerical results. The key idea is to compute Bayes’ rule using a flow of particles rather than as a

point wise multiplication; this solves the well known problem of “particle degeneracy”. Our

derivation is based on freshman calculus and physics. This talk is for normal engineers who do not

have log-homotopy for breakfast.

likelihood of

measurementprior

densityoptimal

accuracyg h

root cause of

curse of dimensionality:

curse of dimensionality:

particles to represent the prior

pdf pdf

particles particles

flow of density

flow of particles

sample from

density

sample from

density

λ=0 λ=1

prior posterior

)(log)(log),(log xhxgxp λλ +=

),( λλ

xfd

dx=

f.for PDE above thesolving

by flow particle design the We

loglog)(

+−=

λd

Kdhppfdiv

35

method to solve PDE how to pick unique solution comments

1. generalized inverse of linear differential operator minimum norm* very difficult to design robust stable & fast algorithm

2. Poisson’s equation gradient of potential*(assume irrotational flow)

very difficult to design robust stable & fast algorithm

3. generalized inverse of gradient of log-homotopy assume incompressible flow & pick minimum L² norm solution

workhorse for multimodal densities

4. generalization of method #3 most robustly stable filter or random pick, etc.

workhorse for multimodal densities

5. separation of variables (Gaussian) pick solution of specific form (polynomial) extremely fast & hard to beat in accuracy for many problems

6. separation of variables

(exponential family)

pick solution of specific form (finite basis functions)

needs theoretical work & numerical experiments

7. variational formulation (Gauss & Hertz) convex function minimization needs work

8. optimal control formulation convex functional minimization (e.g., least action like Monge-Kantorovich)

very high computational complexityaction like Monge-Kantorovich)

9. direct integration (of first order linear PDE in divergence form)

choice of d-1 arbitrary functions should work with enforcement of neutral charge density & importance sampling

10. generalized method of characteristics more conditions (e.g., small curvature or specify curl, or use Lorentz invariance)

needs theoretical work & numerical experiments

11. another homotopy (inspired by Gromov’s

h-principle) like Feynman’s QED perturbation

initial condition of ODE &

uniqueness of sol. to ODE

needs theoretical work & numerical experiments

12. finite dimensional parametric flow

(e.g., f = Ax+b with A & b parameters)

non-singular matrix to invert needs numerical experiments

13. Fourier transform of PDE (divergence form of linear PDE has constant coefficients!)

minimum norm* or most stable flow very difficult to design robust stable & fast algorithm

14. small curvature flow & assumed prior density solve d x d system of linear equations new in 2012. Beats other methods for difficult nonlinear problems.

15. small curvature flow & homotopy for inverse of A + B (sum of two linear operators)

numerically integrate ODE new in 2012. extremely cool theory.

16. small curvature flow & homotopy for generalized inverse of A + B (sum of two linear operators)

numerically integrate ODE new in 2012

36

exact flow filter is many orders of magnitude faster per

particle than standard particle filters

- - - - -

bootstrap

EKF proposal

incomp flow

exact flow

3

104

105

106

107

Med

ian

Co

mp

uta

tio

n T

ime

fo

r 30 U

pd

ate

s (

sec)

d = 30

d = 20

d = 10

d = 5 bootstrap

particle filter

EKF proposal

* Intel Corel 2 CPU, 1.86GHz, 0.98GB of RAM, PC-MATLAB version 7.7

25 Monte Carlo trials10

210

310

410

510

-1

100

101

102

103

Number of Particles

Med

ian

Co

mp

uta

tio

n T

ime

fo

r 30 U

pd

ate

s (

sec)

37

exact flow

EKF proposal

incompressible

flow

particle flow filter is many orders of magnitude faster

real time computation (for the same or better

estimation accuracy)

3 or 4 orders of

3 or 4 orders of magnitude faster

per particle

avoids bottleneck in

many orders of

magnitude faster

3 or 4 orders of magnitude

fewer particles

bottleneck in parallel

processing due to resampling

38

10-1

100

101

102

EKF

PF Incompressible

new filter improves accuracy by

two orders of magnitude

median error

N = 500 particles

extended Kalman filter

standard particle filter

0 2 4 6 8 1010

-3

10-2

10

Time (sec)

PF Incompressible

PF Ax+BN = 500 particles

new filter

key idea: particle flow inspired by fluid

dynamics to make solution of PDE

div(pf) = η much faster

Euler’s equations:

3121233

2313122

1232311

)(

)(

)(

MIII

MIII

MIII

=−+

=−+

=−+

ωωω

ωωω

ωωω

&

&

&

39