a short course by: dr. w. david bell dr. c. david brown
TRANSCRIPT
What T&E’rs Need to Know
About
SYSTEMS ENGINEERING
A n d
PROGRAM MANAGEMENT
And W hy
A Webinar Based On
A Short Course By:
Dr. W. David Bell
Dr. C. David Brown
This Webinar Highlights Selected Topics from
Our 3 Day Short Course
• Overview of Program/Project Management (PM)
• Overview of Systems Engineering (SE)
• Test and Evaluation (T&E) and Interaction with PM and SE
• Modeling and Simulation & T&E
• Current Trends in T&E – T&E early in programs
– Verify vs Validate = DT vs OT
– Integrated T&E
– T&E of immature technologies
– Agile acquisition T&E implications
• Case Studies - Army Future Combat Systems & Navy Littoral Surveillance Radar System
• DOD Acquisition – Process, Life Cycle, Regulations, Directives, Guidance, & Instructions
• Special Topics – Software SE and T&E
– Human Systems Integration
– Unmanned and Autonomous Sys SE and T&E
– Cyber and Info Assurance SE and T&E
31 May 2013, 2100 2
Dave Bell Principal Multi-Discipline Systems Engineer
MITRE Corp
Things done: • Data Analysis • Flight test direction - DT and OT • Scientific experiments • Systems engineering from beginning to end
– Requirements definition – Tech development, tech maturing – I&T at all levels - assembly, subsystem & system
• Managed S&T programs • VP of operations • Adjunct Professor /Lecturer of systems engineering – SMU, JHU
Education • BS Physics, MBA, D. Engineering
31 May 2013, 2100 3
Dave Brown Consulting Systems Engineer
MITRE Corp and Institute for Defense Analyses
Things done: • Colonel, US Army Signal Corps
• Developing DT instrumentation and test facilities
• Development of M&S for test support – Army Virtual Proving Ground
• Live Fire T&E
• Army Senior Executive – Director of Test and Technology, Army Developmental Test Command
• Deputy PM/Executive Director for Test, Army Future Combat Systems Program
• Instructor – JHU Masters in Systems Engineering
Education • BS, MS, PhD Electrical Engineering
• Masters in National Security Policy, ICAF
31 May 2013, 2100 4
Bottom Line Up Front
What PM elements do T&E’rs need to understand? – Leadership, authority
– Stakeholders
– Risk management
– Planning, monitoring, and control
– Work breakdown structure (WBS)
– Budgeting
– Scheduling
– Earned Value Management (EVM)
Why: – All of the above involve some information for which T&E is the source
– Accomplishing T&E involves work (WBS), resources (Budget), and time (Schedule)
– Information from T&E is critical to stakeholder communication, risk management, monitoring, and assessing work completion for EVM
– T&E planning, resourcing, execution, and completion must be “program managed”
C
31 May 2013, 2100 5
Bottom Line Up Front What SE elements do T&E’rs need to understand?
– Needs and definition
– Requirements analysis
– Synthesis
– Integration
– Maturity/Readiness
– Trades
– Verification
– Validation
– Risk management
Why: – All of the above involve some information for which T&E is the source
– The test system must be system engineered
– Early T&E involvement in programs involves T&E and SE integration
– Pushed by GAO and proven by successful programs
C
31 May 2013, 2100 6
Bottom Line Up Front
And most important: – T&E’rs must have a thorough knowledge and
understanding of the total system and system management
process to understand the information and the information
needs and perspectives of each stakeholder,
So that:
– T&E can provide each stakeholder with information that is:
•Required
•Relevant
•Complete
•Accurate
•Appropriate
•Understandable
•In the Correct Context
C
31 May 2013, 2100 7
Our Challenging World
• Terrorism
• Global culture clashes
• Information overload
• Disease – Health Care
• Infrastructure
• Technical Complexity
• Business and Finance
Problem Attributes
• Technical
• Size - Scope
• Complexity
• Importance
• Social, political, fiscal
Systems Engineering and Management is increasingly the
field expected to solve the problems.
C
31 May 2013, 2100 8
9
A Systems Approach
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
C
Data Collection
Mission Performance Analysis
Operational Data Collection
Lessons Learned
Test & Evaluation
Product Development & Production
Prototype Development
Performance Demonstration
Critical Field Experiments
Enabling Science & Technology
Hypothesis, Concept Development Trade-offs & Critical Experiments
Modeling and Simulations
Capabilities Improvement
Needs Definition
Technology Knowledge Transfer
Technology
Operations
Knowledge
31 May 2013, 2100 9
System Management
System Design
& Engineering
Interface
Management
Analysis &
Evaluation
Integration
& Test
Technical Performance,
Cost, Schedule
Data Management
Project Administration
& Support
Configuration
Management
Quality Control
Contract
Management
Production
Management
Systems
Engineering Program/Project
Management
Test &
Evaluation
C
31 May 2013, 2100 10
Senator John McCain, 15 December 2011
To be clear, the military-industrial-congressional complex does not cause programs to
fail. But, it does help create poorly-conceived programs — programs that are so
fundamentally unsound that they are doomed to be poorly executed. And, it does help
keep them alive—long after they should have been ended or restructured.
By “poorly conceived”, I mean major programs that are allowed to begin despite
having insufficiently defined requirements; unrealistic cost or schedule estimates;
immature technology or too much manufacturing and integration risk; or unrealistic
performance expectations.
By “poorly executed”, I am referring to programs that poorly perform because of,
among other things, unanticipated design, engineering, manufacturing or technology
problems. These sorts of programs should either never have been started to begin
with or should have been significantly restructured or terminated at the end of the
day.
Specifically, the military-industrial-congressional complex helps ensure that poorly-
conceived programs get on rails—and stay there—with “production money” when
they are supposed to still be in development. And, for Industry and many of their
sponsors in the Pentagon and on the Hill, that’s desirable because it is far more
difficult to restructure or terminate a production program—even one that’s
performing poorly—than one that’s in development.
Honorable Frank Kendall (USD(AT&L)), ITEA Journal, March
2013
31 May 2013, 2100 12
The purpose of developmental testing is simple; to provide data to program leadership so that good
decisions can be made as early as possible. I have a sign outside my office that is a quote from G.
Edwards Deming: “In God we trust, all others must bring data.” It is our developmental testers
who “bring the data” that is needed to make sound decisions during product development.
Programs are organized in various ways, but whatever the specific organizational model, testing is
the source of the crucial information that provides feedback to program management, chief
engineers, lead system engineers, integrated product teams, and military users on whether their
designs meet requirements or not.
Five precepts – DT&E should: Contribute to program efficiency and effective execution, Provide
relevant information as early as possible, Integrate DT&E planning across the product life cycle,
Focus on support to internal program decisions and verification of compliance with requirements,
and Use DT&E to improve the efficiency and validity of OT&E.
We sometimes fail to conduct adequate DT&E prior to the decision to start production. About a
year ago, I called a particular decision to enter production on an aircraft program without flight
testing “acquisition malpractice.” If a product enters production before the design is stable, the
resulting waste in cost increases and schedule slips can be dramatic, and the program is much more
likely to be canceled. I stress solid, well defined DT&E results as an important prerequisite for this
decision because the pressure to enter production can be overwhelming, and doing so prematurely
has major consequences.
Working with program and engineering leadership as key members of the management team,
developmental testers provide the information that makes program success possible and much
more probable.
“The Test and Evaluation (T&E) process is an integral part
of the Systems Engineering (SE) process, which identifies
levels of performance and assists the developer in correcting
deficiencies. It is a significant element in the decision-making
process, providing data that support trade-off analysis, risk
reduction, and requirements refinement.”*
Both SE and T&E are integral parts of the System/Program
Management Process where SE provides the technical
direction and T&E provides information to support the
technical elements above as well as management processes
and information to stakeholders of the System/Program
Management Process. These include PMs, decision makers,
and Congress.
*DAU Test and Evaluation Management Guide
C
31 May 2013, 2100 13
Form
− Incorporates Many Capabilities
− Comprised of Interacting, Diverse Elements
− Definable Structure and Interconnections
− Bounded; Has Inputs and Outputs
Function
− It Does Something Valuable
− It is Dynamic—Not Inert
Interfaces
− Internal
− External
Attributes View of a Complex System
31 May 2013, 2100 14
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
Each system can be broken down into components which in turn can
be broken down into smaller components, etc.
System
Subsystem
Component Component
Subsystem
Component Component
Subcomponent Subcomponent
Part Part
Air Defense System
Search Radar
T/R Module
Screw
Assembly
Partitioning View of a Complex System
31 May 2013, 2100 15 This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
Systems Perspectives View
Systems Thinking Systems Engineering Engineering Systems
Focus on Process Focus On Whole Product Focus on Both Process and Product
Consideration of Issues Solve Complex Technical
Problems Solve Complex Interdisciplinary
Technical, Social and Management
Issues
Evaluation of Multiple
Factors and Influences Develop and Test Tangible
System Solutions Influence Policy, Processes and use
Systems Engineering to Develop
Systems Solutions
Inclusion of Patterns
Relationships, and Common
Understanding
Need to Meet Requirements,
Measure Outcomes and Solve
Problems
Integrate Human and Technical
Domain Dynamics and Approaches
31 May 2013, 2100 16
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
Systems Management
Key Issues
• SE and T&E have traditionally been very separate processes accomplished by widely separate communities
• SE and PM require information to accomplish knowledge-based development and acquisition
• T&E provides information, but it must be the right information, at the right time, fully understood, and correctly used
• The Test setup is a system that must be System Engineered.
• The T&E program must be Program Managed.
C
31 May 2013, 2100 17
What is a System?
“A System is a set of interrelated components working together toward some common objective.” - (Kossiakoff, et al) The organization of hardware, software, materials, facilities, personnel, data and services needed to perform a designated function with specified results...(Defense Acquisition University) A bounded process involving specific interactions among a number of separately distinguishable functional elements (Amsler, JHU/APL)
31 May 2013, 2100 18
Systems Scope Relationships
Examples
Complexity
Scale
Device – Detector
Component - Sensor
Subsystem - IR Seeker
System – STD Missile
System of Systems - AEGIS BMD
Enterprise Systems – BMD Community
Global Family of Systems -
Theater Defense
31 May 2013, 2100 19
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
A System Includes…
Benjamin S. Blanchard, System
Engineering Management
31 May 2013, 2100 20
The
System
Elements
The System Boundry
System Environment
Benjamin S. Blanchard, System
Engineering Management
31 May 2013, 2100 21
Definition of Systems
Engineering (SE)
• An iterative and interdisciplinary management and development process that defines and transforms requirements into an operational system.
• Features: Typically, this process involves environmental, economic, political, social, and other non-technological aspects. Activities include conceiving, researching, architecting, utilizing, designing, developing, fabricating, producing, integrating, testing, deploying, operating, sustaining, and retiring system elements.
• Note: The customer for or user of the system usually states the initial version of the requirements. Systems engineering should be used to better define and refine these requirements. Often the requirements change as further decisions are made.
31 May 2013, 2100 22
System Engineering Identifying
Qualities
• Top down - viewing system as a whole.
• Life cycle view.
• “Complete” effort to identify system
requirements “up-front”.
• Interdisciplinary team approach.
31 May 2013, 2100 23
Scientific Method
Problem
Definition
Select
Hypothesis
Test Hypothesis
Problem
Next Problem
Conclude
Confirm, Deny or
Modify Hypothesis
31 May 2013, 2100 24
Systems Engineering Process (Method) (Kossiakoff, et al)
1. Requirements
Analysis
2. Functional
Definition
Need
Solution(s)
4. Design
Validation
Requirements
Functions
Potential Solutions
(System Models)
4 Basic SE Activities: executed
repetitively over life cycle
leading to successfully
developed and validated
system
Problem
Definition
3. Physical
Definition or
Design Synthesis
31 May 2013, 2100 25
26
Systems Engineering Approaches
Views
Waterfall
Spiral
The “V” Linear
Life cycle
Systems
Engineering
Lifecycle
Concept
Development
Engineering
Development
Post
Development
Operational
Deficiencies
Technological
Opportunities
System Functional
Specifications
Defined System
Concept
Production
Specifications
Production
System
Operation &
Maintenance
Documentation
Installed Operational
System
Systems
Engineering
Method Requirements
Analysis (Problem Definition)
Functional
Definition (Functional Analysis &
Allocation)
Physical
Definition (Synthesis, Physical Analysis
& Allocation)
Design
Validation (Verification, Evaluation)
Next
Phase
Objectives
Requirements
Functions
System
Model
Validated
System
Model
P&D
E&MD
D&V
CE/D
Need
1
2
3
4
5
Quality
Mgt
Users - Operators
Market Pull
Customer Requirements
Concept
New Product Idea
Technology Push
Preliminary System/Product
Concept Definition
Market Assessment
RFP/| BAA
Discussions
Collaboration
Brainstorming
War rooms
“Draw a Cartoon”
Proposal
Statement of Work
Product Defn Statement
Functional/System
Block Diagram
Pricing/Estimating
WIN! Form Project Office
START WORK
Work breakdown
Structure WBS
Risk Assessment
Plan
Budget and Schedules
PERT and GANTT charts
Task Work Orders
Work Authorizations Develop Prototype
Evaluate Prototype
“beta tests”
Specs
Design
Linear Responsibility Charts
Critical Path Analysis
Contracting
Production
Quantities
Sub System Fabrication
PDR CDR
System Integration
and Verification
System Test and Evaluation Configuration
Mgmt
Field Test and
Evaluation
Operations and
Maintenance
Project Closeout
Follow-On?
Delivery
Install/
Acceptance
Logistics
Warehousing
Sales
NEEDS ANALYSIS CONCEPT EXPLORATION
CONCEPT/PROGRAM DEFINITION
DESIGN / TECHNOLOGY VALIDATION / ENGINEERING DEVELOPMENT PRODUCTION / MANUFACTURING
T/E AND OPERATIONAL SUPPORT SYSTEM USE
Systems Engineering LIFE CYCLE ACTIVITIES
9/20/01 SJSeymour
“PLANNING”
“DIRECTION, MONITOR,CONTROL”
Organizational Structures
Project Mgr Attributes/Authority
Technical Performance
Budget Schedule
Systems
Engineering
Communications and Financial Management All the Time
Slide adapted from a brief by:
Dr. Samuel Seymour,
JHU Applied Physics Lab
31 May 2013, 2100
Life Cycle or Linear Approach to
Systems Engineering Phase
Step
Requirements
Analysis
Functional
Definition
Physical
Definition
Design
Validation
Needs
Analysis
Concept
Exploration
Concept
Definition
Technology
Validation
Engineering
Design Integration
& Evaluation
Analyze
needs
Analyze
operational
reqmts
Analyze
performance
reqmts
Analyze
functional
reqmts
Analyze
design
reqmts
Analyze
reqmts
Define
system
functions
Define
subsystem
functions
Define
component
functions &
interactions
Define
subcomponent
functions
Define
part
functions
Define
functional
tests
Visualize
subsystems,
technology
Visualize
components,
architectures
Select
components,
architectures
Specify
component
construction
Specify
subcomponent
construction
Specify
test
equipment
Validate
needs,
feasibility
Simulate,
validate
performance
reqmts
Simulate,
validate
system
effectiveness
Test
critical
subsystems
Validate
component
design &
construction
Test &
evaluate
production
system
T&E is a key part of every phase
31 May 2013, 2100 27
Motivation for Better
Systems Management
Benjamin S. Blanchard, System
Engineering Management
31 May 2013, 2100 28
The Role of a Systems Engineer
• The technical “leader” and “conductor”
– Sets the objectives (mission needs, system requirements)
– Establishes the plan
– Oversees its execution
– Monitors and guides progress
– Coordinates all technical activities
– Enforces requirements
– Is the final judge of performance/capabilities demonstrated
– Works with Program Manager to orchestrate resources
– Makes the difficult technical decisions
– Manages resolution of technical problems
– Adjusts the plan as necessary
– Advises the Program Manager
– Ultimately responsible for the technical product
31 May 2013, 2100 29
Systems Enterprise • System-of-Systems (SOS) - Combination of
interrelated systems working with direct interaction to achieve a broader range of activities, with the systems generally fixed and invariant. Ex: Ballistic Missile Defense System
• Family-of-Systems (FOS) – Loosely coupled or non-interacting combination of systems intended to perform a broad function where the systems likely share common components and may or may not directly interact. Ex: Stryker Family of Systems
• System of Systems Architecture - The fundamental organization of a SOS, embodied in its systems, their relationships to each other and to the environment and the principles guiding its design and evolution. Ex: Joint Information Environment (JIE), DoD Information Enterprise Architecture (IEA), Global Information Network Architecture (GINA)
31 May 2013, 2100 30
System of Systems Engineering
• Traditionally, SE is focused on the development and implementation of a well-bounded and firmly established set of requirements through the development and integration of subsystems that meet those requirements.
• SoSE leverages independent, interoperable, and integrateable systems that serve a meaningful purpose in a stand-alone mode, to implement operationally flexible capabilities by building a collaborative SoS within an evolving concept of operations to effectively respond to an evolving threat.
• Interoperable = systems that communicate and cooperate
• Integratable = elements that interface in a specified manner to deliver specific functionality or capability
31 May 2013, 2100 31
What makes SoS Engineering
Different?
• Capability focused (e.g., engineering a capability)
• Generally does not have specified requirements
• Focus is more on the architecture than the systems
• Focus is more on interoperability among systems
• Individual systems treated as potentially interchangeable
elements
• More difficult to test - especially at full-up level in lab
• Test challenge – Individual system contribution to SOS
• However…similar engineering practices apply – e.g., SE
method – Integrate and test
31 May 2013, 2100 32
What is a Project/Program/Task?
• Four Characteristics that Distinguish Projects from Other Managerial Activity: – A Three Dimensional Objective (cost, schedule, performance)
– Uniqueness
– Use of Resources
– Accomplishment by an Organization
• Aspects of Projects That May Affect their Difficulty: – Origin of the Project
– Product of the Project
– Marketplace or Customer
– Size, Location
31 May 2013, 2100 33
Project Planning and Control Work Description
And Instructions Network Scheduling
Goals / Objectives
Management
Decision-Making
Master / Detailed
Schedules
Budgets
System Reports
Time / Cost /
Performance
Tracking
0
10
20
30
40
50
60
70
80
90
100
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Continuous Process
31 May 2013, 2100 34
Project Planning
• Identifies and Communicates:
• The Actions to be Taken to Achieve the Project
Objectives. (Performance – WBS)
• The Sequence and Timing of These Actions
Necessary to Achieve the Objectives Efficiently.
(Schedule)
• The Resources ($$) Required to Support the Actions
to Achieve the Objectives. (Budget)
The Triple Constraint Leads to Three
Interrelated Aspects of the Project Plan
31 May 2013, 2100 35
The SE Way
• Customer develops requirements
• PM leads and manages process
• SE guides the
development
process
• T&E informs the
process
• Customer gets the stuff
31 May 2013, 2100 36
The DOD Way
• DOD Acquisition has two customers – The Warfighter/User
– The US Taxpayer/Congress
• Requirements from the User – T&E to Verify/Validate
• Cost and Schedule accountable to the Taxpayer/Congress – T&E to Understand Risks –
Programmatic and Technology
31 May 2013, 2100 37
How the Military Works
31 May 2013, 2100 38
Requirements
Deploy
Secretary of Defense
Service Secretaries
Develop
Build
Buy
39 31 May 2013, 2100
What Happens to the SE “V”
40 31 May 2013, 2100
The PM
What Happens to the SE “V”
More Stakeholders
DOD Acquisition – “The Rules”
31 May 2013, 2100 41
Title 10 USC
Armed Forces
Federal
Acquisition
Regulation
(FAR)
Defense
Acquisition
System
DOD 5000
Service Acquisition Regulations, Instructions, Guidance
Contracting Missile
Defense
Army Marines Air Force Navy
National Defense Authorization Acts (NDAA)
• National Defense Authorization Act is the name of a United
States federal law that has been enacted for each of the past 48
fiscal years to specify the budget and expenditures of the
United States Department of Defense.
• Often additional provisions that deal with Defense
development and acquisition
– 1982 – Nunn-McCurdy Amendment, designed to curtail cost growth
in American weapons procurement programs
– 1984 – DOT&E
– 1996 - Clinger-Cohen Act (CCA), designed to improve the way the
federal government acquires, uses and disposes information technology
– 2009 – Weapon Systems Acquisition Reform Act (WSARA)
31 May 2013, 2100 42
DOD Acquisition
• The DOD has three principal decision-making support systems associated with military acquisition: – Planning, Programming, Budgeting and Execution
(PPBE) Process - Process for strategic planning, program development, and resource determination.
– Joint Capabilities Integration and Development System (JCIDS) - The systematic method established by the Joint Chiefs of Staff for assessing gaps in military joint warfighting capabilities and recommending solutions to resolve these gaps.
– Defense Acquisition System - The management process used to acquire weapon systems and automated information systems, the operation of which is described in DOD 5000 regulation, instruction, guidance.
31 May 2013, 2100 43
• User Articulates the Needs (Sometimes)
• JCIDS Process Develops Requirements (ICD/CDD)
• Acquisition Community Develops Contract SOW with Contractor (SE Executer)
• PPBE Process Provides the Funding
• Acquisition Community Conducts T&E (DT&E) Informed Reviews (Programmatic and Technical) of SE Progression (PDR, CDR, DAES, MS-C)
• Service/Agency OTAs and DOT&E Conduct Final Acceptance T&E (IOT&E) & Report to Congress (Representing the US Taxpayer)
• User Gets the Stuff
31 May 2013, 2100 44
The DOD Way
Operation of the Defense Acquisition System
First Acquisition Framework in 1971
FULL-
SCALE
DEVELOPMENT
Full-Scale
Development
Decision
Program
Initiation
Production
Go-ahead
Decision
CONCEPTUAL
EFFORT PRODUCTION/ DEPLOYMENT
• Decision points: 3
• Phases: 3
• Milestone documents: 1 (Decision Coordinating Paper (DCP))
31 May 2013, 2100 45
The Defense Acquisition Management System
DODI 5000.02*
Relationship to JCIDS
Initial Capabilities
Document (ICD)
Capability Development
Document (CDD)
Capability Production
Document (CPD)
• The Materiel Development Decision precedes entry into
any phase of the acquisition management system
• Entrance Criteria met before entering phase
• Evolutionary Acquisition or Single Step to Full Capability
IOC: Initial Operational Capability
FOC: Full Operational Capability
PDR: Preliminary Design Review
CDR: Critical Design Review
FRP: Full Rate Production
IOC B A
Technology Opportunities & Resources
Materiel Solution Analysis
FRP Decision Review
FOC
Materiel Development Decision
User Needs
PDR CDR
CDD
CPD
ICD
AoA
Pre-Systems Acquisition Systems Acquisition Sustainment
Post CDR Assessment
PDR
Technology
Development
Production &
Deployment
Operations &
Support
Engineering and
Manufacturing Development
Post PDR Assessment
C
or
31 May 2013, 2100 46
* DOD Instruction 5000.02, Operation of
the Defense Acquisition System,
December 8, 2008, USD AT&L
47 31 May 2013, 2100
What Happens to the Development & Acquisition Process
The Forgotten Part of the Process
31 May 2013, 2100 48
D octrine
O rganization
T raining
M ateriel
L eadership
P ersonnel
R esources
The Entire Solution
For a Capability
Controlled/Developed by Service Chiefs
Controlled/Developed by
Service Secretaries
Big $
Big Industry
Big Politics
Systems Engineering - Ideal
Time
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Validate
Verify
31 May 2013, 2100 49
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Systems Engineering – How it Happens
Time
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Validate
Verify
Time
31 May 2013, 2100 50
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Systems Engineering – The Result
Time
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Validate
Verify
Time
31 May 2013, 2100 51
Requirements
Implementation
Integration
& Test
Full System
Test
Time
Validate
Verify Design
31 May 2013, 2100 52
Systems Engineering – How it Should Be
V&V Implications
• DT is more heavily focused around verification
– Among other uses, DT is a main player in deciding to
accept items from contractors
• OT, the focus is both were the requirements met
and are they still the right requirements
– This is a source of friction with PM’s as they believe
they should only be judged by the requirements as
stated not as currently needed.
31 May 2013, 2100 53
54
• Over the past 10 years, DoD systems have experienced a 33% cost growth due to “RDT&E mistakes”…
• DoD IOT&E results, FY2001-2006 – 29 systems; mix of ACAT II, 1C, 1D across 3 Services – Approx. 50% were deemed “Not Suitable”, or partially NS – Approx. 33% were deemed “Not Effective”, or partially NE
• Preliminary study conducted by DOT&E (July 2007) determined that lack of Suitability is a significant life cycle sustainment cost driver – Reliability is the main component – Study revealed a strong correlation between reliability growth (mostly
Testing and M&S) program investments and reductions in support costs
From: Dave Castellano, Deputy Director,
Assessments and Support ODUSD(ATL)
31 May 2013, 2100
55
…We Don’t Start Them Right • Insufficient requirements analysis and definition at program initiation
– Not tangible, measurable, testable, stable
– User R&M requirements are not underpinned by sound rationale
• Acquisition strategies based on poor technical assumptions, competing budget priorities, and unrealistic expectations
• Budget not properly phased
• Lack of rigorous systems engineering approach
• Schedule realism – success oriented, concurrent, poor estimation and/or planning
• Inadequate test planning – breadth, depth, resources
• Optimistic/realistic reliability growth – not a priority during development
• Inadequate software architectures, design/development discipline, and organizational competencies
• Sustainment/life-cycle costs not fully considered (short-sighted)
From: Dave Castellano, Deputy Director, Assessments and Support ODUSD(ATL)
Red denotes areas in which testers can be especially helpful.
31 May 2013, 2100
56
…We Don’t Manage Them Right • Insufficient trade space
– Resources, schedule, performance, requirements
• Insufficient risk management
• Inadequate IMP, IMS, EVMS
• Most programs lack quantifiable entrance/exit criteria
• Maturing “suitability” (e.g., RAM) is not always a priority
• Maturing “effectiveness” is not always a priority
• Concurrent test program; inadequate scope due to schedule and resource insufficiencies, etc.
• Inadequate OTRR process – no strong DT&E gate prior to IOT&E
• Inadequate government staff; Inexperienced and/or limited contractor staffing
• Poorly defined IPT roles, responsibilities and authority
– Overall poor communications across government and industry staff
From: Dave Castellano, Deputy Director, Assessments and Support ODUSD(ATL)
Red denotes areas in which testers can be especially helpful.
31 May 2013, 2100
57
PMs Avoid Testers
As Long As Possible…
• The Situation: – Gov’t program managers typically limit T&E involvement until
absolutely forced to do so. • Perceived as adding schedule and fiscal expense in a time-period
already under schedule and fiscal stress. – (Cynically) Do not want to ask the question if they might not be able to
stand the answer.
– We’ll make up lost time and fix the problems later in the last few weeks/months of the program known as T&E.
– By forgoing T&E insights the program manager du jour might: • Escape problems on their watch but the DOD does not escape
discovering problems and ultimately fixing them
• Have their program cancelled or restructured therefore not fielding important warfighter capabilities
31 May 2013, 2100
A Word About USC Title X, DOT&E, OT&E, & OTA’s
• Do OTAs & DOT&E Represent the User?
• Do They Conduct and Evaluate “User Testing” - Is OT&E User T&E?
• Involved mostly at end of the process
• Report to:
– User?
– Congress
• What happens when the
User gets the stuff?
31 May 2013, 2100 58
The DOD Way Updated • Is the stuff ready for OT?
• Congress Decides they Shouldn’t Wait Until the End of the
Process
• WSARA 2009
– Create DASD(SE)
– Create DASD(DT&E)
– More Emphasis on DT&E (Theoretically throughout the
process)
– However: Better DT&E does not directly translate to better
informed acquisition and development and certainly does not
enhance user involvement in the process
31 May 2013, 2100 59
A Word About System-of-Systems (SOS)
• SOS is the way almost all military equipment gets used
• But it is the way almost none of it is developed
• Recent activity on SOS T&E (mainly Army)
– Mandates to conduct SOS T&E
– Need to address SOS requirements
– Need to manage development by Program of Programs
– Can’t just integrate individual programs (Army ASA(ALT)
SOS Integration Directorate challenges)
– Interoperability testing and integrated testing are not SOS
testing
31 May 2013, 2100 60
The Future • Can the Pure SE Way Work for DOD?
– Not as Long as the User and Purchaser are Different Customers
• Can a Hybrid Work?
– Does the Current Hybrid Work?
• Congress thinks not
• The GAO thinks not
• The User thinks not
– A Better Hybrid?
• COCOM Driven Rapid Acquisition?
– Working, but……
– COCOM of current ops gets all the attention and resources
– Other COCOMs of possible next conflict get little to nothing
• Budget cutting will dictate:
– Fewer new systems
– More evolution – less revolution
– More discipline and rigor
– Better informed acquisition – T&E
– Better Systems Engineering
31 May 2013, 2100 61
What T&E’rs Need to Know
About
SYSTEMS ENGINEERING
A n d
PROGRAM MANAGEMENT
And W hy
A Webinar Based On
A Short Course By:
Dr. W. David Bell
Dr. C. David Brown