flight test data pipeline - international test and...
TRANSCRIPT
Flight Test
Data Pipeline
Aaron PayneGeorgia Tech Research Institute
Electronic Systems [email protected]
Outline
• Data Pipeline
– The problem
– The tools
– The implementation
• Lessons Learned
– Pipeline/data enablers
– Viewing the results
Analysis Types
• Flight Test Analysis
– Generally event specific
– Detections
• Ranges, sensitivity, accuracy
– Performance
• Processing load, response timing, state transitions
• Developmental Analysis
– Generally aggregate
– Improve performance
– Algorithm development
– Algorithm tuning
Firefly (1553)
ffextractInternal Recorder(s)
Download
Conversion of binary data to engineering
units (CSV)
TSPITrack Data
Instrumentation
Analysis
DetectionEnd to End
TimingAccuracy
System Loading
Flight Test Analysis
Developmental Analysis Example
• Problem: Correlate a large number of diverse tracks primarily
based on kinematics
• Given: A correlator capable of grouping tracks based on
uncertainty based kinematic comparison scores
• Characterize sensors!
Fire Control Radar
Targeting Pod Missile Link 16
Intra-flight Data Link
RWR Jammer
Characterizing Sensors
• 1.5+ Terabytes of Flight Test Data, but not tested with this functionality in mind
• Data needed:
– Track data – preferably many samples across all relevant modes and different target types
– Aircraft position data
– Target truth
• Issues:
– Very little “real” target truth
– Data being recorded in new format, IRIG Chapter 10
Firefly (1553)
Internal Recorder(s)
IRIG
Preparing the Data
Tracks
Aircraft Position
Target Truth
Ethernet Playback Utility• IRIG Chapter 10• pcap
IRIG
WinPcap
IRIG106
CSV, SQLite, HDF5
IRIG Ch 10 recording
Ethernet Packets
Track Data
Preparing the Data
Aircraft Position
Target Truth
Tracks
IRIG
CSV, SQLite, HDF5
IRIG Ch 10 recording
1553 Data
Firefly (1553)
IRIG Ch10 -> FireflyConverter
TSPI
Preparing the Data
Target Truth
Tracks
Aircraft Position
• Truth requires additional assets or
special ranges, i.e. unavailable
• How are the sensors tested/used now?
– Sorties are typically 2-4 ships
– Target one another
• Can we use recorded aircraft position
TSPI as truth?
• Yes, but how do we associate TSPI to
tracks?
Preparing the Data
Target Truth
Tracks
Aircraft PositionTrack Data
PC Correlator
TA
TB
TC
TD
TA
TB
TC
Track Assignment
Existing Track to Track Assignment Tool
TSPIInject TSPI as a track ATSPITSPI A
TD
Inadvertent Pipeline (Single Ship)
IRIGWinPcap
IRIG106
IRIG Ch 10 recording
Ethernet Packets
1553 Data IRIG Ch10 -> FireflyConverter
CSV, SQLite, HDF5
Track Data
TSPI
Inadvertent Pipeline (Flight Group)
Track Data #1
Track Data #2
Track Data #3
Track Data #4
TSPI #1
TSPI #2
TSPI #3
TSPI #4
PC Correlator
Track Data #1
TSPI #4TSPI #3TSPI #2TSPI #1
Track Data #2
TSPI #4TSPI #3TSPI #2TSPI #1
Track Data #3
TSPI #4TSPI #3TSPI #2TSPI #1
Track Data #4
TSPI #4TSPI #3TSPI #2TSPI #1
Run 1
Run 2
Run 3
Run 4
Inadvertent Pipeline (Aggregate)
• Batch process all flights over the
multi-year repository
• Retrieve all track report samples
that were time coincident with
“truth”
• Generate statistics on the
aggregated data
Tool Chain Goals
• Simple operation – add new data, get new results
• Provide an architecture for multiple analyses
• Simple analysis integration
• Simple analysis aggregation
• Avoid redundancy – both in engineer work and data crunching
• Relative analysis programming language flexibility
• Portable
• Leverage existing analysis scripts
Pipeline Functional Layout – Stage 1
Pipeline Controller
• Crawl• Processor• Memory• Filtering
• Pipeline Controller Responsibilities
– Crawl raw data repository
– Send data to correct parser
– Maximize hardware horsepower (threads, etc.)
– Filter data and analyses/conversions
Stage 1
Pipeline Functional Layout – Stage 2
Data Converters
IRIG
Firefly
• Data Converter Responsibilities
– Two Inputs
• File to process
• Results Location
– Results in engineering units and
consumable format (CSV, SQLite, etc.)
– Determine whether data needs to be
analyzed (i.e. only re-run if needed)Time
Synchronize
Common Frame
Stage 2
Pipeline Functional Layout – Stage 3
• Single Ship Analysis Responsibilities
– Inputs
• Raw filename
• Converted data directory
• Results directory
– Results in engineering units and
consumable format (CSV, SQLite, etc.)
– Determine whether data needs to be
analyzed (i.e. only re-run if needed)
Stage 3Single Ship Analyses
Route Plot
Max Detections
Software Loaded
Errors and Utilization
Pipeline Functional Layout – Stage 4
• Multi-Ship Analysis Responsibilities
– Inputs
• Group filenames
• Group converted directory paths
• Results directory
– Results in engineering units and
consumable format (CSV, SQLite, etc.)
– Determine whether data needs to be
analyzed (i.e. only re-run if needed)
– Use common library to find files
Stage 4Multi-Ship Analyses
PC Correlator
Truth Statistics
Pipeline Functional Layout – Stage 5
• Aggregate Analysis Responsibilities
– Inputs
• Data directory
• Results directory
– Find results
– Results in engineering units and
consumable format (CSV, SQLite, etc.)
– Determine whether data needs to be
analyzed (i.e. only re-run if needed)
– Use common library to find files
Stage 5Aggregate Analyses
Sensor Uncertainty
Data Coverage
Failure Prevelance
Use Cases
Pipeline Functional Layout
Pipeline Controller
• Crawl• Processor• Memory• Filtering
Data Converters
IRIG
Firefly
Time Synchronize
Common Frame
Stage 1 Stage 2 Stage 3 Stage 4Multi-Ship Analyses
PC Correlator
Truth Statistics
Single Ship Analyses
Route Plot
Max Detections
Software Loaded
Errors and Utilization
Stage 5Aggregate Analyses
Sensor Uncertainty
Data Coverage
Failure Prevalence
Use Cases
Pipeline Enablers
• Machine consumable ICD
• Common reference frame
• Meta Data
– Input data – Who, What, When, Where, Why, Software Versions,
Hardware Versions, etc.
– Analysis data
• When was it generated?
• What source data was used?
– Choose your data storage formats with this in mind
General Hurdles
• Reviewing script failures/successes
• Managing data (while maintaining flexibility)
– Protecting source data
– Versioning
• What combination of source data and script versions created a set of results
– Comparing results
• User access
• Reviewing the results
– More plots is not always better
– Dynamic/Interactive results
Single Track Uncertainty Analysis
• We started with graphs that look like this
• Excellent for evaluating a particular instance
• Frustrating when you have 5000 to look through
Interactive Plot Attempt #1
• Excel– Build plot template based on data
connection data
– Perl dynamically copies template and sets data connection
• Pros– Easy to modify template
– Data is available for additional computation
• Cons– Sometimes slow
– Plots are limited
Interactive Plot Attempt #2
• Javascript Plotting (specifically High Charts)– Build client side web page (no
server)
– Drag and drop csv data
• Pros– Freedom
– Portable
• Cons– Learning curve
– Can struggle with large data
Recommended Tools
• Pandas – Python Data Analysis Library
• D3.js – Data-Drive Documents
• Dygraphs
• Highcharts
• Luigi
• Tableau
• Cesium
Questions?