challenges in data fusion - pennsylvania state university · the origin of multisensor data fusion...

31
Challenges in Data Fusion: Dirty Secrets, Current State of Technology and a Research Roadmap David L. Hall Associate Dean for Research School of Information Sciences and Technology February 28, 2005

Upload: lymien

Post on 12-Jul-2018

214 views

Category:

Documents


0 download

TRANSCRIPT

Challenges in Data Fusion:Dirty Secrets, Current State of Technology and a

Research Roadmap

David L. Hall

Associate Dean for ResearchSchool of Information Sciences and Technology

February 28, 2005

2

Presentation OutlineBrief introduction to data fusionIdentification of the broken promiseDirty secrets in data fusionA grand tour of the JDL model (L-1 through L-5)

Basic conceptsRecent researchChallenges and opportunities

Dirty secrets revisitedA Draft roadmap

3

The origin of multisensor data fusion

“I say fifty, maybe a hundred horses . . . What do you say, Red Eagle?”

4

Definitions* . . .Sensor Fusion = Data Fusion from Multiple Sensors (same or different sensor types)Data Fusion = Combining information to estimate or predict the state of some aspect of the worldData Fusion Functions:

Data Alignment(spatio-temporal, data normalization, evidence conditioning)Data Association(hypothesize entities)State Estimation & Prediction

(etc.)Platform

(etc.)

Reports

Situation

Cross-Force Relations

Force Structure

UnitTraditional

Focus

* Courtesy : Alan Steinberg

5

Data Fusion Functional Model

The JDL model (1987-91) and the draft revised model (1997)

Level 0 — Sub-Object Data Association and Estimation: pixel/signal level data association and characterization

Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete stateestimation (e.g. target type and ID) and prediction

Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc.

Level 3 — Significance Estimation [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment

Level 4 — Process Refinement: adaptive search and processing (an element of resource management)

Adapted from A. Steinberg

6

JDL Data Fusion Process Model

SourcesHuman

ComputerInteraction

DATA FUSION DOMAIN

Level OSignal

Refinement

Level OneObject

Refinement

Level TwoSituation

Refinement

Level ThreeThreat

Refinement

Level FourProcess

Refinement

Database Management System

SupportDatabase

FusionDatabase

7

Smart Dust Vision

www.sice.umkc.edu/~leeyu/Udic/Groups/SensorNetworksPresentationFINAL_0909.ppt

8

Distributed Sensor NetworksCivilian Applications

Source: Cauligi (Raghu) Raghavendra Wireless Sensor Networks and Applications, http://ceng.usc.edu/~raghu

Spatial and geographical surveillance and monitoring Structures and machinery performance and malfunction monitoring and diagnosis Smart Buildings

Habitat Monitoring Seismic Structural Monitoring

Feng et al., System-Architectures for Sensor Networks: Issues, Alternatives, and Directions, www.cs.ucla.edu/~jessicaf/paper/ICCD_SN.ppt

9

Distributed Sensor NetworksMilitary Applications

Source: www.plansys.com - Sensor Networks for Network-Centric Warfare

Battlefield operations (detecting, locating, tracking, and identifying targets)Situation and context awareness (humans and computational devices)Machinery performance and malfunction monitoring

Source: Daniel Van Hook, MIT, “Dynamic Declarative Networking”, DARPA SensITmeeting, Oct 7-8, 1999

10

The DoD Legacy:Extensive Research Investments

JDL Process modelTaxonomy of AlgorithmsLexiconEngineering Guidelines

Architecture SelectionAlgorithm Selection

Evolving Tool-kitsExtensive Legacy of technical papers, booksTraining MaterialsTest-bedsNumerous prototypes

11

On the Road to Network Centric Warfare

. . .Today. . .Pre-Web. . . . . . Joint / Enterprise

Stovepipe systemsLittle or no interoperabilitySome network connections

Pervasive networksPervasive networksMissionMission--effective apps & applets effective apps & applets Assured, interoperable enterprise services Assured, interoperable enterprise services Dynamically Dynamically composablecomposable architecturesarchitecturesRobust & reliable edge computing Robust & reliable edge computing Accurate, timely & relevant infoAccurate, timely & relevant infoImproved Quality of Service (QOS) with Improved Quality of Service (QOS) with centrally managed infrastructurecentrally managed infrastructure

More networksSome web servicesVarious directory & security servicesUncoordinated Service/Functional transformationsFew authoritative data sources

EnterpriseServices

ForceSustainment

Providers

12

JapanKorea

Hawaii

Okinawa

Guam

GIGGIG--BEBE

DoD Satellite Constellation

Net-Centric Enterprise Services“Increment One”

UA UA

SMTA

IMTA IMTA

ROOTDSA

GlobalDSA

RegionalDSA

MFI BMTA

MLA

BMTA

MLA

Storage(Tape/SAN/NAS)

ESM

GESPortal

IA/Security

NCES Services

DECC or Equivalent

Locations shown for CONUS DISA DECCS are notional

NCES Services WillBe Located at 5 DECC

Equivalents Plus 15 SDNs (Which Includes

6 Teleport Sites)

Users Will Be Able to Access NCES Services

From Any LocationWorldwide Using

NIPRNet and SIPRNet

StorageGES

Portal

IA/Security

NCESEdge Server

STEP/Teleport

Commercial andMILSATCOM

Commercial andMILSATCOMNCES Operational Concept

13

Transformation from energy to knowledge

The utility of a data fusion system must be measured by the extent to which it supports effective decision making

Energy Signals Data State vectors Labels Knowledge

14

The Promise of Data Fusion: the Vision of the Empowered Information Analyst

Information Information FusionFusion

Environmental models

Correlated, conditioned data

Real-worldknowledgeTextual data

exemplars

Graphic, Applied Computing 4-Day Conference, www.ac-conference.com, 2000.

Enabling technologies exist:- Wideband communications- Ubiquitous high-performance computing

Enabling technologies exist:- Wideband communications- Ubiquitous high-performance computing

15

What We Got: The Current Information Analysis Reality

Challenges to Intelligence Analysis

• Most problems are complex and information rich but model poor

• Required reaction times have decreased from days to minutes

• Fewer analysts are available to work more data and more problems

• Anywhere, anytime threats with no a priori doctrine

• Information gathering equity by 3rd

world and terrorist opponents• AccuWeather

• Commercial satellite imagery

• Commercial communications and computing resources allows technological leap frog

Radical new approaches are required to change the current reality to meet the new vision

16

Challenges in Data Fusion

Robust sensors:no perfect sensors availabledifficult to predict sensor performanceunable to effectively task geographically distributed non-commensurate sensors

Image and non-image fusion:no true fusion of imagery and non-imagery dataunable to optimally translate image in time-series data into meaningful symbolsno requisite models for coherent fusion of non-commensurate sensor data

Robust target identification:insufficient training dataunable to perform automated feature extractionno unified method for incorporating implicit and explicit information for identification (e.g., information learned from exemplars, model information, and cognitive-based contextual information)

17

Challenges in Data Fusion

Unified calculus of uncertainty:do not know how to effectively use these techniques limited experience in trade-offs and use of fuzzy logic, rules probability, Dempster-Shafer’s method, etc.unsure how to select the best uncertainty method

Pathetic cognitive models for Level 2 and 3:unknown how to select the appropriate knowledge representation techniquesargue about competing methodsdo not know how to use hybrid methodsunable to perform knowledge engineering

18

Challenges in Data Fusion

Non-commensurate sensors:uncertainty as how to optimize use of wildly non-commensurate sensorsinability to know how to link decision needs to sensor managementunable to effectively use 10N sensorsno consensus on MOE/MOP

Human computer interface (HCI):trendy and driven by technology and not cognitive needs of usersuffer from the Gutenberg Bible syndromeno effective tools to overcome cognitive deficienciesunable to capitalize on built-in human pattern recognition (e.g., recognition of faces, concepts of harmony)

19

Dirty secrets in data fusion (2000)

There is no substitute for a good sensor.Downstream processing cannot absolve the sins of upstream processing.The fused answer may be worse than the best sensor.There are no magic algorithms.There will never be enough training data.It is difficult to quantify the value of data fusion.Fusion is not a static process.

20

A Grand Tour of the JDL Model

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level OneObject

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

21

A Grand Tour of the JDL Model: Sensors and Level 0 Processing

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level OneObject

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

SourcePre-Processing

Level OneObject

Refinement

Level OneObject

Refinement

Level TwoSituation

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

LevelFive

Cognitive Refinement

Recent Advances

• Ubiquitous nano-scale, smart, low-cost sensors• Embedded signal & image processing• Increased agility in tasking & sensor operation• Improved modeling of sensor performance and signal propagation

Advanced sensors and improved processing at the sensor level provide the opportunity for ubiquitous sensing and accurate modeling of sensor performance

Challenges and opportunities

• Establishment and tracking of pedigree• Common representation of sensor performance• Web-based plug & play service approach• Human carried sensors• Web-service advanced models for sensor performance & propagation• Counter-measures & information warfare

22

A Grand Tour of the JDL Model: Level 1 Processing – target tracking

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level OneObject

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

SourcePre-Processing

Level OneObject

Refinement

Level OneObject

Refinement

Level TwoSituation

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

LevelFive

Cognitive Refinement

Recent Advances

• Continued evolution of random set theoretic methods• Incorporation of identify information to resolve correlation ambiguities• Advanced methods in multiple hypothesis tracking• Distributed processing implementations

Target tracking is becoming increasingly mature, evolving towards increased accuracy and sophistication

Challenges and opportunities

• Increasing sophistication in move-stop-hide strategies• Relatively rapid movement of targets• Tracking of individual humans• Generalized concepts of target & entity• New algorithms for addressing correlation & interdependent sensor measurements

23

A Grand Tour of the JDL Model: Level 1 Processing – target ID

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level OneObject

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

SourcePre-Processing

Level OneObject

Refinement

Level OneObject

Refinement

Level TwoSituation

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

LevelFive

Cognitive Refinement

Recent Advances

• Advances in implicit pattern recognition techniques such as neural nets and cluster algorithms• High performance computing allows physics based modeling of observable features • Hybrid methods incorporate both implicit and explicit information

Target identification transforms observed attribute data into a label or declaration of target identity

Challenges and opportunities

• Increased sophistication in multi-INT spoofing• Relatively rapid movement of targets• New user-centric approaches to target identification and information aggregation• Improved observation prediction using high-performance target models

24

A Grand Tour of the JDL Model: Level 2 & 3 Processing – target tracking

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level OneObject

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

SourcePre-Processing

Level OneObject

Refinement

Level OneObject

Refinement

Level TwoSituation

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

LevelFive

Cognitive Refinement

Recent Advances

• Utilization of agent-based technologies• Fuzzy logic reasoning• Hybrid implicit/explicit reasoning techniques• Incorporation of semantic & numerical information• Maturation of Bayes & D-S Belief Nets

Level-2 & Level-3 fusion is very challenging; it involves the attempt to emulate human reasoning

Challenges and opportunities

• What is a target now anyway?• Emerging techniques for image representation via semantic concepts• Leveraging of new search engines• Advances in cognitive modeling

25

A Grand Tour of the JDL Model: Level 4 Processing

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level OneObject

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

SourcePre-Processing

Level OneObject

Refinement

Level OneObject

Refinement

Level TwoSituation

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

LevelFive

Cognitive Refinement

Recent Advances

• Ubiquitous nano-scale, smart low-cost sensors• Wide-bandwidth communications• Web-services approach for “plug-in” services• Advances in control theory• TRIP model

Ubiquitous sensing, wide bandwidth communications & distributed processing provide both opportunities & challenges forsensor and process control & optimization

Challenges and opportunities

• E-business models for auction-based resource optimization• Multi-time scale sensors & processing requirements• Human-in-the-loop optimization• Utilization of agent-based methods • Need for multi-INT validation

26

A Grand Tour of the JDL Model: Level 5 Processing

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level OneObject

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

SourcePre-Processing

Level OneObject

Refinement

Level OneObject

Refinement

Level TwoSituation

Refinement

Level TwoSituation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Database Management System

SupportDatabase

FusionDatabase

Level ThreeThreat

Refinement

Level ThreeThreat

Refinement

LevelFive

Cognitive Refinement

LevelFive

Cognitive Refinement

Recent Advances

• 3-D displays• Haptic interfaces• Gesture recognition & NLP• Computer aided cognition & collaboration• Agent models of team interaction

New HCI technology and cognitive models provide opportunities for enhanced effectiveness of the human-fusion system

Challenges and opportunities

• Overwhelming data from “live” feeds• Cognitive limitations and biases• Collaboration from multiple experts (different expertise, cultures, decision styles)• Rapid technology advances from the e-world (affective computing, adaptive interfaces)

27

Dirty Secrets Revisited (2005)Dirty Secrets (2005)

There is still no substitute for a good sensor (and a good human to interpret the results)Downstream processing still cannot absolve upstream sinsNo only may the fused result be worse than the best sensor –but failure to address pedigree, information overload, and uncertainty may really screw up thingsStill no magic algorithms (yes, even agents, ontologies, D-S nets, etc.)Never enough data (but we can help by hybrid methods)We’ve started at the wrong end (and continue to focus on the wrong end)It’s not the data!!!

Dirty Secrets (2000)There is no substitute for a good sensor.Downstream processing cannot absolve the sins of upstream processing.The fused answer may be worse than the best sensor.There are no magic algorithms.There will never be enough training data.It is difficult to quantify the value of data fusion.Fusion is not a static process.

28

Penn State Data Fusion Research

Sources

National

Distributed

Local

Initial EW SONAR RADAR

.

.

.DATA BASES

Sensor Design- Adaptive Arrays- Synthetic Aperture- Matched Field/Mode- Robust Arrays

Detection/Classification- WB and NB- Wavelets- Signal Design- Adaptive DSP

Advanced Pattern RecognitionMHT

Decision Theory

Distributed DBMS- Adaptive User Interfaces- Schema Models- Network Modeling/PA

Hybrid Methods for Computer

Optimization- Neural Networks- Genetic

Algorithms

Advanced Propagation

Analysis

3-D Simulation Virtual Reality

Approximate Reasoning- Fuzzy Logic- Continuous Interference Net- Logical Templates- Hybrid Reasoning

29

Operational Concept for NGA

Information Sphere• Selected images• Semantic labels• Interpretation layers• Context spatial information

Analyst Support Functions• Search of images using semantic labels• Find “like” images• Context-based interpretation of images• Multi-analyst collaboration aids• Anomaly/event detection• Analyst/information interface tools

S1

S2

SN

••

HUMINTOpen-Source

Automated Semantic Labeling

Semantic Labels

Text-based &parametric searches

Context-based Reasoning

Multi-modal interaction

Operational concept provides a basis for improved geospatial intelligence analysis by “conservation of analyst attention”, enabling the analyst to focus on analysis and interpretation versus information searching (both of current/archival data and system “memory”)

30

A Draft Roadmap

Prediction methods

Massively parallel machine learning of rare events & activities

Event/activity centric agents for evidence aggregation

Hybrid methods for implicit/explicit PR New data mining & search techniques

Automated methods to conduct simultaneous mining, archival, and hypothesis evaluation

New methods for representation of hierarchal & generalized events, activities & entities

Characterization of rare events & activities

Improve detection & discovery of “rare” events

Curmudgeon agentsCognitive aides to mitigate biases, etc.

Improved performance of human-in-the-loop decision maker (s) Machine

directed focus of attention

Mixed (human & intelligent S/W) cognition tools & environmentsMOE/MOP develop.

Decision support toolsStress reduction Collaboration aids (team-based agents)

Address multi-timescale phenomenaDevelop auction-based methods for resource allocation

New models of uncertainty

Development of problem-centered decomposition methodsModel pedigree

Model human sources & performance

Unacknowledged sourcesImproved information source modelsImproved use of information resources

Affective computing HCIMulti-dimensional displays & multi-modal I/FMulti-level I/F

Haptic I/FUser-adaptive I/F (model user-behavior)Dialog-based I/FNew search engines

Semantic language-based dialog systemsVisualization of non-physical phenomenaImproved interface between the user & fusion system

Automated ontology generationHigh level ontology development

Merging auto-ontologies with hierarchical reasoning

Agent interpretersUse of hierarchical reasoning techniquesNew displays

Full-up data to semantic hierarchical reasoning

Semantic integration of images & signalsAutomated semantic labeling of signal data

Automated semantic labeling of images

Improved ability to understand data

Year 5Year 4Year 3Year 2Year 1Desired Capability

31

SummaryRapid evolution of sensors, wide-band communications and data proliferation provides opportunities and challengesEffective approaches must focus on human-centric analysis and improvementsRapid evolution of information technology provides leverage opportunities