challenges in data fusion - pennsylvania state university · the origin of multisensor data fusion...
TRANSCRIPT
Challenges in Data Fusion:Dirty Secrets, Current State of Technology and a
Research Roadmap
David L. Hall
Associate Dean for ResearchSchool of Information Sciences and Technology
February 28, 2005
2
Presentation OutlineBrief introduction to data fusionIdentification of the broken promiseDirty secrets in data fusionA grand tour of the JDL model (L-1 through L-5)
Basic conceptsRecent researchChallenges and opportunities
Dirty secrets revisitedA Draft roadmap
3
The origin of multisensor data fusion
“I say fifty, maybe a hundred horses . . . What do you say, Red Eagle?”
4
Definitions* . . .Sensor Fusion = Data Fusion from Multiple Sensors (same or different sensor types)Data Fusion = Combining information to estimate or predict the state of some aspect of the worldData Fusion Functions:
Data Alignment(spatio-temporal, data normalization, evidence conditioning)Data Association(hypothesize entities)State Estimation & Prediction
(etc.)Platform
(etc.)
Reports
Situation
Cross-Force Relations
Force Structure
UnitTraditional
Focus
* Courtesy : Alan Steinberg
5
Data Fusion Functional Model
The JDL model (1987-91) and the draft revised model (1997)
Level 0 — Sub-Object Data Association and Estimation: pixel/signal level data association and characterization
Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete stateestimation (e.g. target type and ID) and prediction
Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc.
Level 3 — Significance Estimation [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment
Level 4 — Process Refinement: adaptive search and processing (an element of resource management)
Adapted from A. Steinberg
6
JDL Data Fusion Process Model
SourcesHuman
ComputerInteraction
DATA FUSION DOMAIN
Level OSignal
Refinement
Level OneObject
Refinement
Level TwoSituation
Refinement
Level ThreeThreat
Refinement
Level FourProcess
Refinement
Database Management System
SupportDatabase
FusionDatabase
8
Distributed Sensor NetworksCivilian Applications
Source: Cauligi (Raghu) Raghavendra Wireless Sensor Networks and Applications, http://ceng.usc.edu/~raghu
Spatial and geographical surveillance and monitoring Structures and machinery performance and malfunction monitoring and diagnosis Smart Buildings
Habitat Monitoring Seismic Structural Monitoring
Feng et al., System-Architectures for Sensor Networks: Issues, Alternatives, and Directions, www.cs.ucla.edu/~jessicaf/paper/ICCD_SN.ppt
9
Distributed Sensor NetworksMilitary Applications
Source: www.plansys.com - Sensor Networks for Network-Centric Warfare
Battlefield operations (detecting, locating, tracking, and identifying targets)Situation and context awareness (humans and computational devices)Machinery performance and malfunction monitoring
Source: Daniel Van Hook, MIT, “Dynamic Declarative Networking”, DARPA SensITmeeting, Oct 7-8, 1999
10
The DoD Legacy:Extensive Research Investments
JDL Process modelTaxonomy of AlgorithmsLexiconEngineering Guidelines
Architecture SelectionAlgorithm Selection
Evolving Tool-kitsExtensive Legacy of technical papers, booksTraining MaterialsTest-bedsNumerous prototypes
11
On the Road to Network Centric Warfare
. . .Today. . .Pre-Web. . . . . . Joint / Enterprise
Stovepipe systemsLittle or no interoperabilitySome network connections
Pervasive networksPervasive networksMissionMission--effective apps & applets effective apps & applets Assured, interoperable enterprise services Assured, interoperable enterprise services Dynamically Dynamically composablecomposable architecturesarchitecturesRobust & reliable edge computing Robust & reliable edge computing Accurate, timely & relevant infoAccurate, timely & relevant infoImproved Quality of Service (QOS) with Improved Quality of Service (QOS) with centrally managed infrastructurecentrally managed infrastructure
More networksSome web servicesVarious directory & security servicesUncoordinated Service/Functional transformationsFew authoritative data sources
EnterpriseServices
ForceSustainment
Providers
12
JapanKorea
Hawaii
Okinawa
Guam
GIGGIG--BEBE
DoD Satellite Constellation
Net-Centric Enterprise Services“Increment One”
UA UA
SMTA
IMTA IMTA
ROOTDSA
GlobalDSA
RegionalDSA
MFI BMTA
MLA
BMTA
MLA
Storage(Tape/SAN/NAS)
ESM
GESPortal
IA/Security
NCES Services
DECC or Equivalent
Locations shown for CONUS DISA DECCS are notional
NCES Services WillBe Located at 5 DECC
Equivalents Plus 15 SDNs (Which Includes
6 Teleport Sites)
Users Will Be Able to Access NCES Services
From Any LocationWorldwide Using
NIPRNet and SIPRNet
StorageGES
Portal
IA/Security
NCESEdge Server
STEP/Teleport
Commercial andMILSATCOM
Commercial andMILSATCOMNCES Operational Concept
13
Transformation from energy to knowledge
The utility of a data fusion system must be measured by the extent to which it supports effective decision making
Energy Signals Data State vectors Labels Knowledge
14
The Promise of Data Fusion: the Vision of the Empowered Information Analyst
Information Information FusionFusion
Environmental models
Correlated, conditioned data
Real-worldknowledgeTextual data
exemplars
Graphic, Applied Computing 4-Day Conference, www.ac-conference.com, 2000.
Enabling technologies exist:- Wideband communications- Ubiquitous high-performance computing
Enabling technologies exist:- Wideband communications- Ubiquitous high-performance computing
15
What We Got: The Current Information Analysis Reality
Challenges to Intelligence Analysis
• Most problems are complex and information rich but model poor
• Required reaction times have decreased from days to minutes
• Fewer analysts are available to work more data and more problems
• Anywhere, anytime threats with no a priori doctrine
• Information gathering equity by 3rd
world and terrorist opponents• AccuWeather
• Commercial satellite imagery
• Commercial communications and computing resources allows technological leap frog
Radical new approaches are required to change the current reality to meet the new vision
16
Challenges in Data Fusion
Robust sensors:no perfect sensors availabledifficult to predict sensor performanceunable to effectively task geographically distributed non-commensurate sensors
Image and non-image fusion:no true fusion of imagery and non-imagery dataunable to optimally translate image in time-series data into meaningful symbolsno requisite models for coherent fusion of non-commensurate sensor data
Robust target identification:insufficient training dataunable to perform automated feature extractionno unified method for incorporating implicit and explicit information for identification (e.g., information learned from exemplars, model information, and cognitive-based contextual information)
17
Challenges in Data Fusion
Unified calculus of uncertainty:do not know how to effectively use these techniques limited experience in trade-offs and use of fuzzy logic, rules probability, Dempster-Shafer’s method, etc.unsure how to select the best uncertainty method
Pathetic cognitive models for Level 2 and 3:unknown how to select the appropriate knowledge representation techniquesargue about competing methodsdo not know how to use hybrid methodsunable to perform knowledge engineering
18
Challenges in Data Fusion
Non-commensurate sensors:uncertainty as how to optimize use of wildly non-commensurate sensorsinability to know how to link decision needs to sensor managementunable to effectively use 10N sensorsno consensus on MOE/MOP
Human computer interface (HCI):trendy and driven by technology and not cognitive needs of usersuffer from the Gutenberg Bible syndromeno effective tools to overcome cognitive deficienciesunable to capitalize on built-in human pattern recognition (e.g., recognition of faces, concepts of harmony)
19
Dirty secrets in data fusion (2000)
There is no substitute for a good sensor.Downstream processing cannot absolve the sins of upstream processing.The fused answer may be worse than the best sensor.There are no magic algorithms.There will never be enough training data.It is difficult to quantify the value of data fusion.Fusion is not a static process.
20
A Grand Tour of the JDL Model
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
Level OneObject
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
21
A Grand Tour of the JDL Model: Sensors and Level 0 Processing
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
Level OneObject
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
SourcePre-Processing
Level OneObject
Refinement
Level OneObject
Refinement
Level TwoSituation
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
LevelFive
Cognitive Refinement
Recent Advances
• Ubiquitous nano-scale, smart, low-cost sensors• Embedded signal & image processing• Increased agility in tasking & sensor operation• Improved modeling of sensor performance and signal propagation
Advanced sensors and improved processing at the sensor level provide the opportunity for ubiquitous sensing and accurate modeling of sensor performance
Challenges and opportunities
• Establishment and tracking of pedigree• Common representation of sensor performance• Web-based plug & play service approach• Human carried sensors• Web-service advanced models for sensor performance & propagation• Counter-measures & information warfare
22
A Grand Tour of the JDL Model: Level 1 Processing – target tracking
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
Level OneObject
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
SourcePre-Processing
Level OneObject
Refinement
Level OneObject
Refinement
Level TwoSituation
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
LevelFive
Cognitive Refinement
Recent Advances
• Continued evolution of random set theoretic methods• Incorporation of identify information to resolve correlation ambiguities• Advanced methods in multiple hypothesis tracking• Distributed processing implementations
Target tracking is becoming increasingly mature, evolving towards increased accuracy and sophistication
Challenges and opportunities
• Increasing sophistication in move-stop-hide strategies• Relatively rapid movement of targets• Tracking of individual humans• Generalized concepts of target & entity• New algorithms for addressing correlation & interdependent sensor measurements
23
A Grand Tour of the JDL Model: Level 1 Processing – target ID
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
Level OneObject
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
SourcePre-Processing
Level OneObject
Refinement
Level OneObject
Refinement
Level TwoSituation
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
LevelFive
Cognitive Refinement
Recent Advances
• Advances in implicit pattern recognition techniques such as neural nets and cluster algorithms• High performance computing allows physics based modeling of observable features • Hybrid methods incorporate both implicit and explicit information
Target identification transforms observed attribute data into a label or declaration of target identity
Challenges and opportunities
• Increased sophistication in multi-INT spoofing• Relatively rapid movement of targets• New user-centric approaches to target identification and information aggregation• Improved observation prediction using high-performance target models
24
A Grand Tour of the JDL Model: Level 2 & 3 Processing – target tracking
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
Level OneObject
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
SourcePre-Processing
Level OneObject
Refinement
Level OneObject
Refinement
Level TwoSituation
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
LevelFive
Cognitive Refinement
Recent Advances
• Utilization of agent-based technologies• Fuzzy logic reasoning• Hybrid implicit/explicit reasoning techniques• Incorporation of semantic & numerical information• Maturation of Bayes & D-S Belief Nets
Level-2 & Level-3 fusion is very challenging; it involves the attempt to emulate human reasoning
Challenges and opportunities
• What is a target now anyway?• Emerging techniques for image representation via semantic concepts• Leveraging of new search engines• Advances in cognitive modeling
25
A Grand Tour of the JDL Model: Level 4 Processing
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
Level OneObject
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
SourcePre-Processing
Level OneObject
Refinement
Level OneObject
Refinement
Level TwoSituation
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
LevelFive
Cognitive Refinement
Recent Advances
• Ubiquitous nano-scale, smart low-cost sensors• Wide-bandwidth communications• Web-services approach for “plug-in” services• Advances in control theory• TRIP model
Ubiquitous sensing, wide bandwidth communications & distributed processing provide both opportunities & challenges forsensor and process control & optimization
Challenges and opportunities
• E-business models for auction-based resource optimization• Multi-time scale sensors & processing requirements• Human-in-the-loop optimization• Utilization of agent-based methods • Need for multi-INT validation
26
A Grand Tour of the JDL Model: Level 5 Processing
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
Level OneObject
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
HumanComputerInteraction
Sources
DATA FUSION DOMAIN
Level FourProcess
Refinement
SourcePre-Processing
SourcePre-Processing
Level OneObject
Refinement
Level OneObject
Refinement
Level TwoSituation
Refinement
Level TwoSituation
Refinement
Database Management System
SupportDatabase
FusionDatabase
Database Management System
SupportDatabase
FusionDatabase
Level ThreeThreat
Refinement
Level ThreeThreat
Refinement
LevelFive
Cognitive Refinement
LevelFive
Cognitive Refinement
Recent Advances
• 3-D displays• Haptic interfaces• Gesture recognition & NLP• Computer aided cognition & collaboration• Agent models of team interaction
New HCI technology and cognitive models provide opportunities for enhanced effectiveness of the human-fusion system
Challenges and opportunities
• Overwhelming data from “live” feeds• Cognitive limitations and biases• Collaboration from multiple experts (different expertise, cultures, decision styles)• Rapid technology advances from the e-world (affective computing, adaptive interfaces)
27
Dirty Secrets Revisited (2005)Dirty Secrets (2005)
There is still no substitute for a good sensor (and a good human to interpret the results)Downstream processing still cannot absolve upstream sinsNo only may the fused result be worse than the best sensor –but failure to address pedigree, information overload, and uncertainty may really screw up thingsStill no magic algorithms (yes, even agents, ontologies, D-S nets, etc.)Never enough data (but we can help by hybrid methods)We’ve started at the wrong end (and continue to focus on the wrong end)It’s not the data!!!
Dirty Secrets (2000)There is no substitute for a good sensor.Downstream processing cannot absolve the sins of upstream processing.The fused answer may be worse than the best sensor.There are no magic algorithms.There will never be enough training data.It is difficult to quantify the value of data fusion.Fusion is not a static process.
28
Penn State Data Fusion Research
Sources
National
Distributed
Local
Initial EW SONAR RADAR
.
.
.DATA BASES
Sensor Design- Adaptive Arrays- Synthetic Aperture- Matched Field/Mode- Robust Arrays
Detection/Classification- WB and NB- Wavelets- Signal Design- Adaptive DSP
Advanced Pattern RecognitionMHT
Decision Theory
Distributed DBMS- Adaptive User Interfaces- Schema Models- Network Modeling/PA
Hybrid Methods for Computer
Optimization- Neural Networks- Genetic
Algorithms
Advanced Propagation
Analysis
3-D Simulation Virtual Reality
Approximate Reasoning- Fuzzy Logic- Continuous Interference Net- Logical Templates- Hybrid Reasoning
29
Operational Concept for NGA
Information Sphere• Selected images• Semantic labels• Interpretation layers• Context spatial information
Analyst Support Functions• Search of images using semantic labels• Find “like” images• Context-based interpretation of images• Multi-analyst collaboration aids• Anomaly/event detection• Analyst/information interface tools
S1
S2
SN
••
HUMINTOpen-Source
Automated Semantic Labeling
Semantic Labels
Text-based ¶metric searches
Context-based Reasoning
Multi-modal interaction
Operational concept provides a basis for improved geospatial intelligence analysis by “conservation of analyst attention”, enabling the analyst to focus on analysis and interpretation versus information searching (both of current/archival data and system “memory”)
30
A Draft Roadmap
Prediction methods
Massively parallel machine learning of rare events & activities
Event/activity centric agents for evidence aggregation
Hybrid methods for implicit/explicit PR New data mining & search techniques
Automated methods to conduct simultaneous mining, archival, and hypothesis evaluation
New methods for representation of hierarchal & generalized events, activities & entities
Characterization of rare events & activities
Improve detection & discovery of “rare” events
Curmudgeon agentsCognitive aides to mitigate biases, etc.
Improved performance of human-in-the-loop decision maker (s) Machine
directed focus of attention
Mixed (human & intelligent S/W) cognition tools & environmentsMOE/MOP develop.
Decision support toolsStress reduction Collaboration aids (team-based agents)
Address multi-timescale phenomenaDevelop auction-based methods for resource allocation
New models of uncertainty
Development of problem-centered decomposition methodsModel pedigree
Model human sources & performance
Unacknowledged sourcesImproved information source modelsImproved use of information resources
Affective computing HCIMulti-dimensional displays & multi-modal I/FMulti-level I/F
Haptic I/FUser-adaptive I/F (model user-behavior)Dialog-based I/FNew search engines
Semantic language-based dialog systemsVisualization of non-physical phenomenaImproved interface between the user & fusion system
Automated ontology generationHigh level ontology development
Merging auto-ontologies with hierarchical reasoning
Agent interpretersUse of hierarchical reasoning techniquesNew displays
Full-up data to semantic hierarchical reasoning
Semantic integration of images & signalsAutomated semantic labeling of signal data
Automated semantic labeling of images
Improved ability to understand data
Year 5Year 4Year 3Year 2Year 1Desired Capability