dr. steven j. hutchison principal deputy developmental test and...

Shift Left Nov 2012 Page-1

DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395

Dr. Steven J. Hutchison Principal Deputy

Developmental Test and Evaluation November 2012



Persistent, rapidly composable, secure representation of the Joint Information Environment

Test & Evaluation

Operations

Performance Reliability

DT&E for Complex Systems

System Integration Labs

Training

Experimentation

Modeling & Simulation

Cyber Range

JIOR

JMETC

Interoperability Information Security



The DoD Acquisition Model



Test, Evaluation, Certification

Late to Need!

DIACAP Security T&E



Hindsight is 20-20

What did we know?

What did we test?

To reduce discovery late in the acquisition lifecycle, • test in mission context, • against realistic threat,

and….. Shift Left!

DOT&E COCOM/Service

Interop & IA Assessments

Fielded systems: • Interoperability issues • IA Vulnerabilities

Compliance with IA Controls and

Interoperability Standards and Profiles

are necessary but not sufficient



Net Ready KPP New Role for DASD(DT&E)

New Language • “DISA will ensure JITC leverages

previous, planned and executed DT&E and OT&E tests and results to support joint interoperability test certification and eliminate test duplication.”

• “DASD(DT&E) shall approve Developmental Test and Evaluation plans in support of Joint Interoperability Test Certification as documented in the TEMP. JITC shall advise DASD (DT&E) regarding the adequacy of test planning in support of Joint Interoperability Test Certification.”

DASD(DT&E) approves adequacy of Interoperability test planning

CJCSI 6212



Information Assurance Policy

Information Assurance compliance activities need to be integrated into DT&E and included in the TEMP



Information Assurance What’s Changing?

• Implements Risk Management Framework (RMF) instead of Mission Assurance Category/Confidentiality Level (MAC/CL)

• Adopts new guidance from the National Institute of Standards and Technology (NIST) and Committee on National Security Systems Instruction (CNSSI) documents on Cybersecurity

• Goes beyond IA and adopts the term: “Cybersecurity”

• Lexicon Changes – “Certification and Accreditation” becomes “Assessment and Authorization” – “Designated Approving Authority (DAA)” becomes “Authorizing Official (AO)” – “Certifying Authority” becomes “Security Control Assessor”

Threat = Any event with potential to cause harm to the network Vulnerability = Absence/weakness of safeguards to protect the network

Risk = Likelihood that a threat will realize or exploit a vulnerability



Implementing Cybersecurity What’s Being Proposed?

DASD(DT&E):

• Oversight of test planning in support of Cybersecurity C&A(A&A)

• Establish procedures to ensure that DT&E authorities for acquisition programs verify that adequate DT&E is planned and resourced to address Cybersecurity

• Confirm DT&E can be executed in a timely manner prior to approval of program Test and Evaluation Master Plans (TEMPs)

DASD(DT&E) will ensure adequate Cybersecurity test planning



DT&E in the Cyberspace Domain

An Integrated T&E Enterprise Capable of Creating a Realistic Cyberspace Test Environment at All Required

Security Levels

Cyberspace Threat Representations

Systems Under Test

Test Tools

Instrumentation BAF

JPRIMES

ACETEF

CDS

IO Range

SDREN

TSMO

Desired Federated Cyberspace T&E Capability

Process

Methodology Infrastructure

Workforce

Persistent, rapidly composable, secure representation of the Joint Information Environment



DT&E Cybersecurity Process Summary

Step 1 Cybersecurity Test

Requirements Evaluation

Focus on initiating an approach to Cybersecurity DT&E at Milestone A or B, with update at Milestone C.

Step 4 Cybersecurity Test in

Realistic Cyber Environment

Focus is on Cybersecurity readiness in an operational mission environment to understand capabilities and limitations of the SUT and interconnections against a cyber threat using Red Team testing.

Step 3 Cyber Kill Chain

Evaluation

Focus is assessment of Cybersecurity of the system under test, in a realistic mission and cyber environment, using exploitation testing techniques, post-CDR.

Step 2 Cybersecurity

System Integration Evaluation

Focus is assessment of Cybersecurity in component and system integration vulnerability testing, between MS B and C.



Cybersecurity Testing in the Acquisition Lifecycle

AS TDS

JCIDS Process

Full Rate Production

Decision Review

CBA Joint

Concepts (COCOMs)

MS C MS B

Strategic Guidance (OSD/JCS)

MS A

ICD Technology Development

CDD Engineering & Manufacturing Development

Production and Deployment O&S MDD

Materiel Solution Analysis

AoA

CPD

TEMP

SEP

SRR SFR PDR CDR TRR SVR

*TEMP

*SEP

ASR

PPP

TRA *PPP STAR OTR

TEMP

IOT&E

* STAR * STAR

* SRD

AOTR

Cyber Test Step 1

Cyber Test Step 1 Step 2

Cyber Test Step 1 Step 2 Step 3

Cyber Test Step 1 Step 2 Step 3 Step 4

Reduce the Cyber Attack Surface



Conclusion

• DT&E in mission context

• Improve Interoperability

• Improve Cybersecurity

• Reduce discovery in IOT&E

• Improve Acquisition Outcomes

To ensure rapid fielding of enhanced capabilities to the Warfighter …

Shift Left!



Questions?



T&E Plan – Test – Report cycle can exceed six months!

•Multiple Test Orgs – DT, OT, Iop, IA

•Multiple Decision Makers – MDA, CIO, DAA

Pilot Record OTRR

60 days

OTRR Full Deployment Decision Review

60 days

Eval Report

DIACAP

Interop Testing

OT&E

Operational Test Plan

Test Concept Brief

60 days

Test Plan Approved

User Training Support Implemented

Interop Cert

IAC&A

Tester Training DT&E

14 days

DoD Test, Evaluation, & Certification

AOTR

dr. steven j. hutchison principal deputy developmental test and...

Documents