test & evaluation challenges for the future - itea - …...test & evaluation challenges for...

38
Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11, 2011

Upload: others

Post on 29-May-2020

36 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Test amp Evaluation

Challenges for the Future

Chip Ferguson

Joint Mission Environment Test Capability

Program Manager May 11 2011

2

DoD Strategic Plan

for TampE Resources

Service

TampE Needs and

Solutions Process

Service

TampESampT

Working Groups

Relationship within TRMC

Synergy through Aligned Investment

Quadrennial Defense Review

Strategic Planning Guidance

Service Improvement

amp Modernization

Programs

Acquisition Programs

Advanced Concept

Technology Demonstrations

TampE Multi-Service

Agency

Capabilities

Risk mitigation needs Technology shortfalls

Risk mitigation solutions

Advanced development

Requirements

Capabilities

DoD Corporate

Distributed Test

Capability

TRMC Joint Investment Programs

63 64 65

Distributed Testing

A process preferably persistent and continuous for linking

various geographically separated Live Virtual and

Constructive sites and systems together in a distributed

environment for use across the acquisition life cycle to

support and conduct the Test and Evaluation (TampE) of a

system or systems-of-systems

GOAL Near Real-Time Test-Fix-Test

across organizational boundaries

3

Why Consider Distributed Test

bull Is there a requirement to exchange data within your system or within a

system-of-systems (SoS)

bull Do you have a requirement to address SoS interoperability issues early

in the acquisition process

bull Do you need more systems under test for live testing than what is

available at your facility

bull Do you have all of the appropriate ldquosupporting castrdquo (supporting

systems C4ISR assets etc) needed for testing

bull Do you need additional threat types fidelity and density in realistic

numbers at realistic ranges for testing

bull Do you need to incorporate TampE into training events

4

If you are answering lsquoNOrsquo to any of these

questions you will soon be saying lsquoYESrsquo

When Is Distributed Test Appropriate

bull Examples Where Appropriate

ndash Interoperability testing

bull C4 Interoperability with higher lower and adjacent Joint force organizations

ndash Data exchange in early DT testing

bull Interaction between sub-systems (latency may be a consideration)

bull Interaction between systems in a realistic environment

o Provide the most realistic environment possible from concept exploration through

Follow-On TampE

ndash When it is too costly to bring all the player systems together on a single

range

ndash Gain or increase operational relevance

bull Virtual and Constructive capabilities to supplement live systems (eg red

forces white forces terrain immobile test assets)

bull Examples Where Inappropriate

ndash System performance tests that do not include other systemssubsystems

ndash Reliability testing

5

Reduces Cost Risk and Time

6

JMETC Enabled Distributed Testing

Systems

Under

Test

Joint Operational Scenarios

Virtual

Prototype

Range Environment

Generator

Threat

Systems

Integrated Test

Resources

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

Reuse Repository

Distributed Test Support Tools

JMETC

Infrastructure

Hardware in the Loop Lab

TENA

Common

Middleware

Installed Systems

Test Facility

Basic Interface

Standard

TENA

Common

Middleware

Data Management Solutions

TENA

Standard Interface

Definitions

TENA

Standard Interface

Definitions

JMETC

Network on

SDREN

6

Distributed TampE Supports

Systems Engineering (SE)

Hardware Fabrication

ComponentInterface Definition

Allocated Functions amp Performance Requirements

Requirements Definitions (JCIDS) Transition

Requirements Analysis

Logical Analysis

Design Solution

Integration

Verification

Validation

Implementation

Tech Data amp Training Pubs

Software CreationCoding

Detailed Design

Confirmation Audits

Realization amp Assessment

Decomposition amp Definition

Input Output to Next Phase of

Development or Lifecycle Support Operational Need

Measures of EffectivenessSuitability

System Measures of Performance

Element Design Criteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent

Des

ign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Tech Readiness

Assessment

Early TampE

Involvement

Component Level

Testing

Component

Characterization

Early Operational

Assessment

DTampE

Verification

OTampE

Validation

Source DAU Acquisition Community Connection

Early Injection of

Authoritative

systems and

mission threads

Combined DTampE

OTampE in a Joint

or SoS

Environment

Persistently Integrated

but disparately

developed capabilities

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 2: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

2

DoD Strategic Plan

for TampE Resources

Service

TampE Needs and

Solutions Process

Service

TampESampT

Working Groups

Relationship within TRMC

Synergy through Aligned Investment

Quadrennial Defense Review

Strategic Planning Guidance

Service Improvement

amp Modernization

Programs

Acquisition Programs

Advanced Concept

Technology Demonstrations

TampE Multi-Service

Agency

Capabilities

Risk mitigation needs Technology shortfalls

Risk mitigation solutions

Advanced development

Requirements

Capabilities

DoD Corporate

Distributed Test

Capability

TRMC Joint Investment Programs

63 64 65

Distributed Testing

A process preferably persistent and continuous for linking

various geographically separated Live Virtual and

Constructive sites and systems together in a distributed

environment for use across the acquisition life cycle to

support and conduct the Test and Evaluation (TampE) of a

system or systems-of-systems

GOAL Near Real-Time Test-Fix-Test

across organizational boundaries

3

Why Consider Distributed Test

bull Is there a requirement to exchange data within your system or within a

system-of-systems (SoS)

bull Do you have a requirement to address SoS interoperability issues early

in the acquisition process

bull Do you need more systems under test for live testing than what is

available at your facility

bull Do you have all of the appropriate ldquosupporting castrdquo (supporting

systems C4ISR assets etc) needed for testing

bull Do you need additional threat types fidelity and density in realistic

numbers at realistic ranges for testing

bull Do you need to incorporate TampE into training events

4

If you are answering lsquoNOrsquo to any of these

questions you will soon be saying lsquoYESrsquo

When Is Distributed Test Appropriate

bull Examples Where Appropriate

ndash Interoperability testing

bull C4 Interoperability with higher lower and adjacent Joint force organizations

ndash Data exchange in early DT testing

bull Interaction between sub-systems (latency may be a consideration)

bull Interaction between systems in a realistic environment

o Provide the most realistic environment possible from concept exploration through

Follow-On TampE

ndash When it is too costly to bring all the player systems together on a single

range

ndash Gain or increase operational relevance

bull Virtual and Constructive capabilities to supplement live systems (eg red

forces white forces terrain immobile test assets)

bull Examples Where Inappropriate

ndash System performance tests that do not include other systemssubsystems

ndash Reliability testing

5

Reduces Cost Risk and Time

6

JMETC Enabled Distributed Testing

Systems

Under

Test

Joint Operational Scenarios

Virtual

Prototype

Range Environment

Generator

Threat

Systems

Integrated Test

Resources

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

Reuse Repository

Distributed Test Support Tools

JMETC

Infrastructure

Hardware in the Loop Lab

TENA

Common

Middleware

Installed Systems

Test Facility

Basic Interface

Standard

TENA

Common

Middleware

Data Management Solutions

TENA

Standard Interface

Definitions

TENA

Standard Interface

Definitions

JMETC

Network on

SDREN

6

Distributed TampE Supports

Systems Engineering (SE)

Hardware Fabrication

ComponentInterface Definition

Allocated Functions amp Performance Requirements

Requirements Definitions (JCIDS) Transition

Requirements Analysis

Logical Analysis

Design Solution

Integration

Verification

Validation

Implementation

Tech Data amp Training Pubs

Software CreationCoding

Detailed Design

Confirmation Audits

Realization amp Assessment

Decomposition amp Definition

Input Output to Next Phase of

Development or Lifecycle Support Operational Need

Measures of EffectivenessSuitability

System Measures of Performance

Element Design Criteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent

Des

ign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Tech Readiness

Assessment

Early TampE

Involvement

Component Level

Testing

Component

Characterization

Early Operational

Assessment

DTampE

Verification

OTampE

Validation

Source DAU Acquisition Community Connection

Early Injection of

Authoritative

systems and

mission threads

Combined DTampE

OTampE in a Joint

or SoS

Environment

Persistently Integrated

but disparately

developed capabilities

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 3: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Distributed Testing

A process preferably persistent and continuous for linking

various geographically separated Live Virtual and

Constructive sites and systems together in a distributed

environment for use across the acquisition life cycle to

support and conduct the Test and Evaluation (TampE) of a

system or systems-of-systems

GOAL Near Real-Time Test-Fix-Test

across organizational boundaries

3

Why Consider Distributed Test

bull Is there a requirement to exchange data within your system or within a

system-of-systems (SoS)

bull Do you have a requirement to address SoS interoperability issues early

in the acquisition process

bull Do you need more systems under test for live testing than what is

available at your facility

bull Do you have all of the appropriate ldquosupporting castrdquo (supporting

systems C4ISR assets etc) needed for testing

bull Do you need additional threat types fidelity and density in realistic

numbers at realistic ranges for testing

bull Do you need to incorporate TampE into training events

4

If you are answering lsquoNOrsquo to any of these

questions you will soon be saying lsquoYESrsquo

When Is Distributed Test Appropriate

bull Examples Where Appropriate

ndash Interoperability testing

bull C4 Interoperability with higher lower and adjacent Joint force organizations

ndash Data exchange in early DT testing

bull Interaction between sub-systems (latency may be a consideration)

bull Interaction between systems in a realistic environment

o Provide the most realistic environment possible from concept exploration through

Follow-On TampE

ndash When it is too costly to bring all the player systems together on a single

range

ndash Gain or increase operational relevance

bull Virtual and Constructive capabilities to supplement live systems (eg red

forces white forces terrain immobile test assets)

bull Examples Where Inappropriate

ndash System performance tests that do not include other systemssubsystems

ndash Reliability testing

5

Reduces Cost Risk and Time

6

JMETC Enabled Distributed Testing

Systems

Under

Test

Joint Operational Scenarios

Virtual

Prototype

Range Environment

Generator

Threat

Systems

Integrated Test

Resources

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

Reuse Repository

Distributed Test Support Tools

JMETC

Infrastructure

Hardware in the Loop Lab

TENA

Common

Middleware

Installed Systems

Test Facility

Basic Interface

Standard

TENA

Common

Middleware

Data Management Solutions

TENA

Standard Interface

Definitions

TENA

Standard Interface

Definitions

JMETC

Network on

SDREN

6

Distributed TampE Supports

Systems Engineering (SE)

Hardware Fabrication

ComponentInterface Definition

Allocated Functions amp Performance Requirements

Requirements Definitions (JCIDS) Transition

Requirements Analysis

Logical Analysis

Design Solution

Integration

Verification

Validation

Implementation

Tech Data amp Training Pubs

Software CreationCoding

Detailed Design

Confirmation Audits

Realization amp Assessment

Decomposition amp Definition

Input Output to Next Phase of

Development or Lifecycle Support Operational Need

Measures of EffectivenessSuitability

System Measures of Performance

Element Design Criteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent

Des

ign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Tech Readiness

Assessment

Early TampE

Involvement

Component Level

Testing

Component

Characterization

Early Operational

Assessment

DTampE

Verification

OTampE

Validation

Source DAU Acquisition Community Connection

Early Injection of

Authoritative

systems and

mission threads

Combined DTampE

OTampE in a Joint

or SoS

Environment

Persistently Integrated

but disparately

developed capabilities

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 4: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Why Consider Distributed Test

bull Is there a requirement to exchange data within your system or within a

system-of-systems (SoS)

bull Do you have a requirement to address SoS interoperability issues early

in the acquisition process

bull Do you need more systems under test for live testing than what is

available at your facility

bull Do you have all of the appropriate ldquosupporting castrdquo (supporting

systems C4ISR assets etc) needed for testing

bull Do you need additional threat types fidelity and density in realistic

numbers at realistic ranges for testing

bull Do you need to incorporate TampE into training events

4

If you are answering lsquoNOrsquo to any of these

questions you will soon be saying lsquoYESrsquo

When Is Distributed Test Appropriate

bull Examples Where Appropriate

ndash Interoperability testing

bull C4 Interoperability with higher lower and adjacent Joint force organizations

ndash Data exchange in early DT testing

bull Interaction between sub-systems (latency may be a consideration)

bull Interaction between systems in a realistic environment

o Provide the most realistic environment possible from concept exploration through

Follow-On TampE

ndash When it is too costly to bring all the player systems together on a single

range

ndash Gain or increase operational relevance

bull Virtual and Constructive capabilities to supplement live systems (eg red

forces white forces terrain immobile test assets)

bull Examples Where Inappropriate

ndash System performance tests that do not include other systemssubsystems

ndash Reliability testing

5

Reduces Cost Risk and Time

6

JMETC Enabled Distributed Testing

Systems

Under

Test

Joint Operational Scenarios

Virtual

Prototype

Range Environment

Generator

Threat

Systems

Integrated Test

Resources

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

Reuse Repository

Distributed Test Support Tools

JMETC

Infrastructure

Hardware in the Loop Lab

TENA

Common

Middleware

Installed Systems

Test Facility

Basic Interface

Standard

TENA

Common

Middleware

Data Management Solutions

TENA

Standard Interface

Definitions

TENA

Standard Interface

Definitions

JMETC

Network on

SDREN

6

Distributed TampE Supports

Systems Engineering (SE)

Hardware Fabrication

ComponentInterface Definition

Allocated Functions amp Performance Requirements

Requirements Definitions (JCIDS) Transition

Requirements Analysis

Logical Analysis

Design Solution

Integration

Verification

Validation

Implementation

Tech Data amp Training Pubs

Software CreationCoding

Detailed Design

Confirmation Audits

Realization amp Assessment

Decomposition amp Definition

Input Output to Next Phase of

Development or Lifecycle Support Operational Need

Measures of EffectivenessSuitability

System Measures of Performance

Element Design Criteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent

Des

ign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Tech Readiness

Assessment

Early TampE

Involvement

Component Level

Testing

Component

Characterization

Early Operational

Assessment

DTampE

Verification

OTampE

Validation

Source DAU Acquisition Community Connection

Early Injection of

Authoritative

systems and

mission threads

Combined DTampE

OTampE in a Joint

or SoS

Environment

Persistently Integrated

but disparately

developed capabilities

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 5: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

When Is Distributed Test Appropriate

bull Examples Where Appropriate

ndash Interoperability testing

bull C4 Interoperability with higher lower and adjacent Joint force organizations

ndash Data exchange in early DT testing

bull Interaction between sub-systems (latency may be a consideration)

bull Interaction between systems in a realistic environment

o Provide the most realistic environment possible from concept exploration through

Follow-On TampE

ndash When it is too costly to bring all the player systems together on a single

range

ndash Gain or increase operational relevance

bull Virtual and Constructive capabilities to supplement live systems (eg red

forces white forces terrain immobile test assets)

bull Examples Where Inappropriate

ndash System performance tests that do not include other systemssubsystems

ndash Reliability testing

5

Reduces Cost Risk and Time

6

JMETC Enabled Distributed Testing

Systems

Under

Test

Joint Operational Scenarios

Virtual

Prototype

Range Environment

Generator

Threat

Systems

Integrated Test

Resources

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

Reuse Repository

Distributed Test Support Tools

JMETC

Infrastructure

Hardware in the Loop Lab

TENA

Common

Middleware

Installed Systems

Test Facility

Basic Interface

Standard

TENA

Common

Middleware

Data Management Solutions

TENA

Standard Interface

Definitions

TENA

Standard Interface

Definitions

JMETC

Network on

SDREN

6

Distributed TampE Supports

Systems Engineering (SE)

Hardware Fabrication

ComponentInterface Definition

Allocated Functions amp Performance Requirements

Requirements Definitions (JCIDS) Transition

Requirements Analysis

Logical Analysis

Design Solution

Integration

Verification

Validation

Implementation

Tech Data amp Training Pubs

Software CreationCoding

Detailed Design

Confirmation Audits

Realization amp Assessment

Decomposition amp Definition

Input Output to Next Phase of

Development or Lifecycle Support Operational Need

Measures of EffectivenessSuitability

System Measures of Performance

Element Design Criteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent

Des

ign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Tech Readiness

Assessment

Early TampE

Involvement

Component Level

Testing

Component

Characterization

Early Operational

Assessment

DTampE

Verification

OTampE

Validation

Source DAU Acquisition Community Connection

Early Injection of

Authoritative

systems and

mission threads

Combined DTampE

OTampE in a Joint

or SoS

Environment

Persistently Integrated

but disparately

developed capabilities

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 6: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

6

JMETC Enabled Distributed Testing

Systems

Under

Test

Joint Operational Scenarios

Virtual

Prototype

Range Environment

Generator

Threat

Systems

Integrated Test

Resources

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

TENA Standard Interface

Definitions

TENA

Common

Middleware

Reuse Repository

Distributed Test Support Tools

JMETC

Infrastructure

Hardware in the Loop Lab

TENA

Common

Middleware

Installed Systems

Test Facility

Basic Interface

Standard

TENA

Common

Middleware

Data Management Solutions

TENA

Standard Interface

Definitions

TENA

Standard Interface

Definitions

JMETC

Network on

SDREN

6

Distributed TampE Supports

Systems Engineering (SE)

Hardware Fabrication

ComponentInterface Definition

Allocated Functions amp Performance Requirements

Requirements Definitions (JCIDS) Transition

Requirements Analysis

Logical Analysis

Design Solution

Integration

Verification

Validation

Implementation

Tech Data amp Training Pubs

Software CreationCoding

Detailed Design

Confirmation Audits

Realization amp Assessment

Decomposition amp Definition

Input Output to Next Phase of

Development or Lifecycle Support Operational Need

Measures of EffectivenessSuitability

System Measures of Performance

Element Design Criteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent

Des

ign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Tech Readiness

Assessment

Early TampE

Involvement

Component Level

Testing

Component

Characterization

Early Operational

Assessment

DTampE

Verification

OTampE

Validation

Source DAU Acquisition Community Connection

Early Injection of

Authoritative

systems and

mission threads

Combined DTampE

OTampE in a Joint

or SoS

Environment

Persistently Integrated

but disparately

developed capabilities

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 7: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Distributed TampE Supports

Systems Engineering (SE)

Hardware Fabrication

ComponentInterface Definition

Allocated Functions amp Performance Requirements

Requirements Definitions (JCIDS) Transition

Requirements Analysis

Logical Analysis

Design Solution

Integration

Verification

Validation

Implementation

Tech Data amp Training Pubs

Software CreationCoding

Detailed Design

Confirmation Audits

Realization amp Assessment

Decomposition amp Definition

Input Output to Next Phase of

Development or Lifecycle Support Operational Need

Measures of EffectivenessSuitability

System Measures of Performance

Element Design Criteria

Spec

ialt

y En

gin

eeri

ng

Dev

elo

pm

ent

Des

ign

En

gin

eeri

ng

Dec

isio

n

Au

tho

rity

Tech Readiness

Assessment

Early TampE

Involvement

Component Level

Testing

Component

Characterization

Early Operational

Assessment

DTampE

Verification

OTampE

Validation

Source DAU Acquisition Community Connection

Early Injection of

Authoritative

systems and

mission threads

Combined DTampE

OTampE in a Joint

or SoS

Environment

Persistently Integrated

but disparately

developed capabilities

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 8: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

8

Sustainment Systems Acquisition (Engineering amp manufacturing development

demonstration LRIP amp production)

Pre-Systems Acquisition

IOC A B

Concept amp Tech Development

Concept

Exploration

Component

Advanced

Development

Decision

Review

System

Integration

Interim Progress Review

Production amp Deployment

LRIP

FRP Decision Review

Full-Rate Production amp Deployment

Operations

amp Support

C

The Future of Distributed Testing Test-Fix-Test

Across the Acquisition Life Cycle

Rapid Acquisition Developmental Test Operational

Test Interoperability Certification Net-Ready Key

Performance Parameters testing Joint Mission

Capability Portfolio testing

JMETC enables continuous testing across the acquisition life cycle JMETC reduces acquisition time and cost By Providing

bull Readily-available

persistent connectivity

bull Standing network

security agreements

bull Common

interoperability software

for linking sites

bull Certified test tools for

distributed testing

Enables early verification that

systems work stand alone and in a

Joint Environment

Helps find problems early in

acquisition ndash when they are less

costly to fix

Outline Distributed

Testing and JMETC

requirements in TEMP

Creates robust environment for common prototype analysis

Provides subject matter expertise to integrate distributed facilities

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 9: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Test Early in Acquisition Process

Recent Distributed Test at a major test facility

bull C2 Interoperability upgrade to National Strategic asset

bull Significant software and interoperability problems found during initial Distributed Testing

bull Discovered system NOT ready for open air testing IMPACTmdashRisk Reduction Early

Identification of Problem

bullEarly Distributed Testing led to early

identification and correction of

interoperability issues

bullNo range or flying costs 9

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 10: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Post-Milestone A ndash Concept Development

Joint Surface Warfare JCTD

Description

Point Mugu Test Team

demonstrated Net Enabled

Weapon Link-16 capability

using FA-18EF as launch

platform JSOW C-1 as

weapon and JSTARS as 3rd

party target source

Sample of Measures

Tactical message timing

between the controller

and the weapon

IMPACT-- Efficiency Lower Technical

Risk and Cost Savings

bull Program scheduled and executed short

multiple tests for incremental software

update evaluation

bull Resources expended on test amp analysis and

not network setup and monitoring

China Lake FA-18 JSOW

10

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 11: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

11

Description bull Assessment of B-2 Link-16

interoperability with AWACS

bull Connected live B-2 on ramp

at Whiteman AFB MO an

AWACS HITL at Tinker AFB

OK within a distributed C2

environment

IMPACT-- Cost Savings and Better

Product

bull Early testing led to early identification

and correction of Link 16 interoperability

issues

bull No range or flying costs

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Army

Air Force

Navy

Marines

Joint

Industry

As of 14 Sep 2009

Tinker AFB AWACS

Whiteman B-2

Eglin 46TS

WPAFB SIMAF

JMETC

Langley GCIC

Post-Milestone C ndash Developmental Test

B-2 Spirit ICE Data Link Test

Sample of Measures

Timeliness and accuracy

of message exchanges

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 12: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

bull Joint Urgent Operational Need

bull Integration of BACN payload

onto multiple platforms for

solution to net-centric warfare

challenges of

bull Beyond line-of-sight comm

bull Relay bridge and range

extension for ground forces

and supporting aircraft

bull Distributed Testing included

Live-fly DT and Operational

Utility Evaluation Fall of 2010

IMPACT -- Cost Savings and

Better Product bullEfficient integration of DT and OT

bullAll required assets not available on-site

bullDistributed Testing saved ldquo$12Mrdquo

bullUrgent capability fielded

12

Responsive TampE for Rapid Fielding

Battlefield Airborne Communication Node

(BACN)

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 13: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Challenges We Are Overcoming Today

Challenge

bull Site Interconnectivity

bull No common enabling

transport architecture

bull Inconsistent Tools

bull Test Participant

Collaboration

bull Analysis of Disparately

Stored Data

Overcome By

JMETC Network

TENA Middleware

JMETC ldquoBest of Breedrdquo

Process

Reuse Repository

Data Management Solutions

JMETC Joint Mission Environment Test Capability

TENA Test and Training Enabling Architecture

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 14: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

A TRMC Perspective

We Need a Common Vision for

Test and Training Infrastructure

Mr Derrick Hinton Principal Deputy Director Test Resource Management Center (TRMC)

Deputy Director Joint Investment Programs amp Policy (JIPP)

Office of the Undersecretary of Defense for Acquisition Technology amp Logistics (OUSD[ATampL])

Annual ITEA Test Instrumentation Workshop XIV

Test and Training Enterprise

12 May 2010

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 15: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

2010 2030

Individual Systems WHAT do we test Mission-Level

amp Systems of Systems Capabilities

Events WHEN do we test Continuously

Decision-Makers For WHOM do we test Decision-Makers

amp Warfighters

Agile

Streamlined

Affordable

Continuous

Goal Distributed Integrated Testing and Training

The DoD TampE process must evolve to be

Test Resource Management Center

Future TampE Process

15

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 16: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Aspects of the 2030 Common

Vision

bull Test amp Training bull Real-Time TTP SOP Development to improve SoS effectiveness and

suitability (limited)

bull ldquoBlack boxrdquo and ldquowhite boxrdquo testing done in parallel

including

bull System functionality

bull System effectiveness

bull System of Systems effectiveness via mission threads in an

operationally relevant environment

bull Fewer test runsconfigurations with more meaningful

collected data

bull Real-Time Data Analysis amp Automated Reporting

OBJECTIVE Near Real-Time Test-Fix-Test 16

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 17: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Some Challenges to Achieving

2030 Vision

bull Authoritative Joint Mission threads

bull Mission level measures

bull Thorough but Streamlined Information Assurance

Procedures

bull Enterprise-level Multi-Level Security solutions

bull Embedded Encryption on acquisition systems

bull RF over the wire (network)

bull Near Real Time Test-Fix-Test

bull Data Management

bull Embedded amp Unobtrusive Instrumentation

bull World wide wireless

bull Extremely efficient usage of the frequency spectrum 17

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 18: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Examples Where the Vision Is Starting

to Come Together

bull Joint Integrated Air amp Missile Defense Organization

(JIAMDO) Correlation Decorrelation Interoperability Test

(CDIT) is reducing analysis cycle from 3 months to less

than a month

bull Changes to test configuration between test runs (~30 minutes)

bull Overnight AdditionSubtraction of test runs depending on issues

discovered and data analysis results

bull Real-time impact to test configuration and number of test runs

bull Emphasis on evaluators and tools for detailed data analysis

bull Air-Ground Integrated Layer Exploration (AGILE) bull AGILE provides an LVC capability for acquisition amp testing

bull Analytical Rigor provides decision quality data for decision support

bull Efficient amp collaborative use of Joint analytical resources (in

addition to service capabilities)

bull Sustains and evolves collaborative environment for future use 18

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 19: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Some TRMC Investments for

Achieving Common Vision

bull Central TampE Investment Program (CTEIP)

bull Test and Training Enabling Architecture (TENA)

bull Multi-Level Security-Joint Coalition Network Environment (MLS-

JCNE)

bull InterTEC

bull Result JITC-certified instrumentation usable in all aspects of TampE

bull Common Range Integrated Instrumentation System (CRIIS)

bull TampE SampT Program

bull TENA in a Resource Constrained Environment (TRCE)

bull NST Evaluation Capability Model (NECM)

bull JMETC Program

bull DoD corporate approach for distributed testing 19

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 20: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

What Can You Do to Be Ready for

2030

bull Maintain Information Assurance agreements for all facility

systems

bull Streamline data collection amp analysis at your facility

bull Get comfortable with DoDAF views

bull Look across the RDTampE the community to find common

problems that can be addressed with similar solutions

bull TENA-enable your instrumentation systems and test

command amp control

bull Get involved share your requirements with your Service

representatives and in TRMC users meetings

20

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 21: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

JMETC Users Group

bull Purpose is to focus on technical requirements and solutions relevant to current and future Distributed Testing needs

bull Technical and Management level representatives identify core infrastructure requirements and most importantly resolve issues

bull Quarterly meetings of 250-300 JMETC customers acquisition programs test events ranges LVC sites tools and network providers

21

Last JMETC Users Group Meeting bull June 2011 bull Location Mid-East Coast bull Tracks

bull User Requirements bull Information Assurance

Security bull Data Management bull InterTEC (Current amp Planned) bull Networking

bull An established forum for the Distributed Test Community to

o Identify core infrastructure requirements and use cases

o Identify investigate amp resolve issues

o Identify opportunities to collaborate

o Discuss available solutions tools and techniques

o Share lessons learned

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 22: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Summary

bull We are becoming more efficient in how we execute TampE

bull We must get the biggest and best ldquobang for the buckrdquo that we can

bull TRMC and Services continue TampE investments to enable realization of the 2030 Vision

bull Align TampE (OSD and Services) investments to mitigate test risk areas

bull The Future of TampE will include Distributed Integrated Testing and Training and will enable near real time Test-Fix-Test

22

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 23: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

JMETC Program Points of Contact

23

JMETC Program Manager Chip Ferguson chipfergusonosdmil 703-601-5274

JMETC Principal Deputy PM Bruce Bailey brucebaileyosdmil 703-601-5208

JMETC Lead Operations Planning Marty Arnwine martemasarnwineosdmil 703-601-5215

JMETC Senior Technical Advisor George Rumford georgerumfordosdmil

703-601-5233

JMETC Lead Systems Engineer Ryan Norman ryannormanosdmil

703-601-5277

JMETC Lead Network Engineer Arjuna ldquoAJrdquo Pathmanathan ArjunaPathmanathanosdmil

703-601-5214

JMETC Website wwwjmetcorg

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 24: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Backup Slides

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 25: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

25

FY10 JMETC Events

Customer Event Record Event Dates

Air Force JEFX 10-1Spirit Ice (B2 Data Link Test) October ndash November 2009

Navy BAMS LVC DE October ndash December 2009

Air Force Battlefield Airborne Comm Node (BACN) JUON

(DTOT)

November 2009 to September 2010

Air Force Agile Fire 10-2 January 2010

Air Force JEFX 10-23 FebruaryApril 2010

Joint Joint Surface Warfare JCTD February to September 2010

JS J8JIAMDO Joint Sensor Integration April to September 2010

Air Force B1-B Fully Integrated Data Link Testing April 2010

JFCOM J8489 (TEST WEEK) JCAS Distributed Test June 2010

JIAMDO (Navy Lead) CorrelationDecorrelation Interoperability Test

(CDIT) Integration Events (Continuous)

July to September 2010

Army (Lead) UAS in the National Airspace July to September 2010

Air Force Agile Fire 10-3 August 2010

Joint JITC Joint Interoperability Tests JIT 10-3 amp 11-1

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Joint Strike Fighter (JSF) JIAMDOJoint Track Manager

Brigade Combat Team (BCT)

Modernization

Multi-Function Adv Data Link (MADL) Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run 25

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 26: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

26

JMETC FY10 Accomplishments

JMETC Accomplishments

Supported 88 distinct customer test activities

Expanded network from 38 to 57 sites

ATIN and JTDL Networks transitioned to JMETC

Upgraded JMETC support applications and utilities to TENA R6

DIACAP Tiger Team report completed and recommendations being executed

Enhanced JMETC services and capabilities provided by leveraging InterTEC Services and Industry

Reuse Repository usability improvements

FY10 Example JMETC Customers

Joint Integrated Air amp Missile Defense Organization (JIAMDO)

Broad Area Communications Node (BACN) JUON

B1-B

Broad Area Maritime Surveillance System (BAMS)

Air-Ground Integrated Layer Exploration (AGILE)

Joint Interoperability Test Command (JITC)

Selected Benefits to the DoD

Integrated DT amp OT on a Joint Urgent Operational Need for the warfighter

Maximized usage of theater assets during limited maintenance windows

Improved Joint track information sharing to ensure interoperability of systems in theater operations

Coalition exchange and examination of real-time air picture data

Identification of Air Force Initiatives ready for warfighter transition

Investigated tactical UAS deployment in the National Airspace

Employment of Net-Enabled Weapons

JCAS immediate request amp end-to-end processes ldquoas-isrdquo characterization

Determined distributed system components were not ready for full live integration testing

Executed testing to support system-of-system interoperability certification

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 27: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

27

Customer Event Record Event Dates

Joint JIAMDO CorrelationDecorrelation Interoperability Test (CDIT)

Integration Events (Continuous) OCONUS October 2010

Navy UAS in NAS Runs for Record October 2010

Internal InterTEC Spiral 3 Systems Acceptance Test October - November 2010

Joint JITC Joint Interoperability Tests JIT 11-12345 (Continuous) October 2010 ndash September 2011

Air Force B1-B Fully Integrated Data Link Testing November 2010

Air Force AGILE Fire Phase III JEFX 2011 December 2010 - February 2011

Joint JTRS JPO -- JTRS Ground Mobile Radio January 2011

Navy Broad Area Maritime Surveillance (BAMS) (Continuous) January ndash September 2011

Joint Joint Track Manager Capability Demonstration (Continuous) January ndash September 2011

Joint JSF Initial MampS Interoperability (Continuous) February ndash March 2011

Air Force JSTARS Interoperability Test May 2011

Joint JS J8JIAMDO Joint Sensor Integration June ndash August 2011

FY11 JMETC Events

(More to Come)

Discussions for Future Teaming

Gerald R Ford Class (CVN-78) Global Hawk GATOR

Brigade Combat Team (BCT)

Modernization F-22 FY12 Testing Planned Multi-Mission Maritime Aircraft (MMA)

Each event is normally preceded by 1-3 spirals Connectivity Check Integration and Dry Run

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 28: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

28

JMETC Connectivity

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca (2) JITC JTDL

Redstone (7) SMDC

RTC DTCC GMAN

SED Patriot THAAD FAAD SIV

Charleston (2)IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB SIMAF

As of 16 Mar 2011

0

10

20

30

40

50

60

70

FY06 FY07 FY08 FY09 FY10 FY11

JMETC Site Expansion

Sites

Dedicated trusted connectivity on SDREN (part of the GIG)

Encrypted for Secret ndash System High

DISA-registered IP address space

Active monitoring of network performance

Capable of supporting multiple simultaneous test events Bethpage NGC BAMS

Whiteman B-2

Eglin AFB (4)AOC DTF GWEF KHILS

Hurlburt Field C2DAC

Sites in Gulf Range

Functional Sites 61

New Sites Planned 8

Connection Points to Other Networks 5

Ft Greely CRTC

Site in Alaska

MHPCC

PMRF Bldg 105

Sites in Hawaii

Camp PendletonMCTSSA

China Lake (3)AV-8B FA-18IBAR

Edwards Ridley

Corona NSWC

El Segundo NGC B-2

Sites in SoCal

Point Loma (2)RLBTSSSC-PAC 59140 SSC-PAC HAIPE

Point Mugu (2)ITEC EA-6B

West Agg Rtr

Rancho Bernardo NGC BAMS

Pax River (5)ESTEL E2CD MCLACETEF SAIL ATR JMETC SYSCONEast Agg Rtr

Aberdeen ACCN

Langley (2) C-GIIF TDLITC

JFCOM JSIC

Wallops Island (2) SCSC SSDS

DISA Sky 7

Pentagon WARCAP

Dahlgren (2) CEDLIWSL

Dam Neck CDSA

Newport News NGC VASCIC

Sites in MD DC VA

Norfolk COMOPTEVFOR

McLean MITRE NCEL

Ft Huachuca JTRS

Bethpage NGC BCN

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 29: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

How an Air Force Customer Sees

Distributed Connectivity

Hardware-in-the-Loop (HWIL) Availability

Army

Air Force

Navy

Marines

Joint

Industry

Ft Huachuca JITC

Redstone (3) DTCC GMAN SED

Charleston (2) IPC MEF-MEU

Ft Hood (2) CTSF TTEC

WPAFB

SIMAF

Bethpage NG BAMS

Whiteman B-2

MHPCC

PMRF Bldg 105

Sites in Hawaii

All linked by JMETC

29

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 30: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Challenges Explained in Detail

30

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 31: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Mission Threads and Measures

bull JFCOM J89 leading work on Mission Threads

o Tier 1 Working ndash EWEA Joint Fires TST Integrated Tactical

Warning

o Tier 1 Complete ndash Counter IED and Global Force Management

o Tier 1 2 3 Working ndash JCAS JPR Air amp Missile Defense

bull Joint Test amp Evaluation Methodology amp JTEM Transition

have led Mission Measures work

o Measures Development Standard Operating Procedure (SOP) to

DOTampE (Sep 10 and Jan 11)

o Mission-Based Test amp Evaluation Assessment Process Guidebook

to DOTampE (Mar 11)

o httpsaccdaumiltije (request an account at

httpsaccdauCommunityBrowseraspx)

bull Much work has been done

bull Much work is yet to do 31

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 32: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Information Assurance

bull DIACAP Tiger Team

o Brought the RDTampE community case to the authors of DIACAP

o JMETC is now a non-voting member of the DIACAP Technical

Advisory Group

o Look for revisions to DIACAP in the near future

o NOTE Ryan I am hoping I can set someone up for this on your

panel

bull Much work to be done

o Streamlined but thorough processes

o Automate the processes where feasible

o Automate management of ports and protocols

32

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 33: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Multi-Level Security

bull Three cases

o Classification

o Coalition

o Proprietary data

bull Need an enterprise-level solution

o Reduce approval times

o Pre-defined solutions

bull CTEIP Multi-Level Secure (MLS) JointCoalition Network

Environment project is in requirements determination

bull Personally I would like the solution to be tied to TENA but

I am not driving the requirements

33

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 34: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Double Encryption

bull Many tactical systems have their own encryption for data

o JTRS

o BAMS

o Others

bull SDREN encrypts all data traversing the network

bull If data is encrypted twice what is the effect

o Are there data issues with double encryption

o Is there a packet loss problem with double encryption

o Is there a work-around to avoid double encryption

o Is there a solution for double encryption

34

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 35: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Near Real Time Test-Fix-Test

bull Requires significant planning

o Test objectives ndash what questions are we trying to answer

o Data collection ndash what data do we need to answer those questions

o Test execution ndash SUT scenarios other systems (blue amp red)

environment etc

o Data management ndash TRMC working on requirements

o Data reduction

o Analysis

bull JIAMDO is very close

o Frank Barone helping to understand the possibilities

o Running a second shift to accommodate

o IDrsquoing issues one day and testing fixes the next

35

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 36: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Data Management

bull Analyst should call for the data heshe needs

o Without regard to where it is stored

o Without regard to the format

Wersquove been trying to standardize data formats between Services and

ranges for 30 years ndash without success

bull And receive that data within minutes

bull Marty Arnwine leading an effort to determine community

requirements

o Requirements will be submitted to the CTEIP for consideration

bull Wersquoll never get where we need to be without very efficient

data management

36

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 37: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Imbedded amp Unobtrusive

Instrumentation

bull If we want to provide information to the Warfighter

o We better measure performance of his systems

o That requires that we have instrumentation on his systems

Unobtrusive

Imbedded

Data transmitted wirelessly

bull TRMCrsquos TampESampT Program

o Testing in a Resource Constrained Environment

Demonstration here

o Imbedded Instrumentation

bull CTEIP

o Integrated Network Enhanced Telemetry

o InterTEC

o Net-Centric Test Agent Capability

37

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38

Page 38: Test & Evaluation Challenges for the Future - ITEA - …...Test & Evaluation Challenges for the Future Chip Ferguson Joint Mission Environment Test Capability Program Manager May 11,

Frequency Spectrum

bull Testing for the Warfighter implies world-wide data

collection

bull It also implies wireless data collection since we cannot

interfere with combat operations

bull This implies extremely efficient use of the frequency

spectrum

bull My boss Mr Hinton is next-up to address this issue

38