test & evaluation - gdit test & evaluation master plan (temp) is a living, ... operational...

Skip NavigationGlossary Acronyms Resources Plugins He

D

Test & EvaluationTest & Evaluation

page 1 of 22

Long Description

Title screen for Module 4, Lesson 2: Technical Management, Test & Evaluation containing three photos: Professionals at a meeting, professionals in a training environment, and a teacher relating information.


Introduction

Welcome to the Test & Evaluation lesson. Test & Evaluation (T&E) is a vital part of a comprehensive acquisition program's strategy to reduce risks, measure success, and inform acquisition decisions.

In this lesson you'll be introduced to the following concepts:

• A definition of T&E in the context of an acquisition program

• The types of T&E program offices are required to conduct

• How T&E activities are integrated into the Acquisition Life Cycle Framework (ALF)

To print this lesson, select print.


page 2 of 22


Learning Objectives

Upon completion of this lesson, you will be able to:

• Define Test & Evaluation (T&E)

• Describe the Test & Evaluation Master Plan (TEMP)

• Identify the purpose and characteristics of Developmental Test & Evaluation (DT&E)

• Identify the purpose and characteristics of Operational Test & Evaluation (OT&E)

• Explain the importance of T&E to acquisition decisions

• Identify the role that T&E plays in the systems engineering process


page 3 of 22


Tests are activities conducted to collect data that verifies whether or not a system is designed to specification and meets the user's performance requirements.

Evaluation is the review and analysis of the data collected during tests. Evaluation answers the question, "What does the data mean?" and attempts to assess system performance under operational conditions.

Test & Evaluation (T&E) is the principle tool used to measure progress and success during system development. That data gathered during T&E provides vital support for acquisition decisions.

What is Test & Evaluation?


page 4 of 22


Why Conduct Test & Evaluation?

One of the main reasons to conduct T&E is to reduce program risk.

Example: The Mazda RX8

The Mazda RX8 was built in 2004 to 2006 and advertised a 250 horse power (HP) engine. The listed engine and Mazda's marketing campaign implied a high performance vehicle, which was a big incentive to buyers. Once drivers got the RX8 on the road, however, the car didn't feel like it was producing 250 HP, and many customers complained.

To follow up on those customer complaints, Mazda put the car and its engine through a series of tests, which revealed the engine only produced 239 HP! There were a number of reasons why the 250 HP engine wasn't performing as it should, which might have been revealed and overcome had Mazda conducted the testing before releasing the vehicle. Instead, Mazda ended up offering drivers a refund or exchange on the car.


page 5 of 22

T&E

Test & Evaluation


Test & Evaluation Roles and Responsibilities

The following roles are involved in the T&E activities for every Level 1, non-delegated Level 2, and special oversight program.

Select each role below for more information about that role's T&E responsibilities.


page 6 of 22

Director, Office of Test & Evaluation (OTE)

The Director of OTE:

• Develops and manages DHS T&E policy and procedures• Advises Program Managers (PM) in planning for T&E, developing T&E documentation, and resolving T&E

issues• Reviews the Concept of Operations (CONOPS) and Operational Requirements Document (ORD)• Observes selected DT events• Reviews DT test plans and reports• Participates in program and design reviews and T&E working groups

Director of Operational Test & Evaluation (OT&E)

The Director OT&E:

• Advises PMs in planning for OT&E, developing OT&E documentation, and resolving OT&E issues• Reviews the Mission Needs Statement (MNS), CONOPS, Integrated Logistics Support Plan (ILSP), and ORD• Reviews and approves the TEMP and OT&E plans• Approves the Operational Test Agent (OTA)• Participates in Operational Test Readiness Reviews and OT&E working groups• Observes OT&E events• Reviews the OTA\'s OT&E reports• Writes the letters of assessment of OT&E reports as appropriate

Component Heads/Component Acquisition Executives (CAEs)

The Component Heads/CAEs:

• Comply with DHS T&E policy• Ensure that adequate and timely T&E are performed to support acquisition decisions (especially Acquisition

Decision Event (ADE)-2C and ADE-3)

User/Sponsor

The User/Sponsor defines the system\'s requirements in the MNS and ORD, specifically the Critical Operational Issues (COIs) and Key Performance Parameters (KPPs) which are the focus and direction for OT&E.

Program Manager (PM)

The PM:

• Has overall responsibility for planning and executing a program\'s T&E strategy• Identifies and recommends an OTA for the program (to conduct OT&E)• Provides resources for T&E activities, including funding, test articles, and integrating T&E activities in the

program schedule

Operational Test Agent (OTA)

The OTA may be from the sponsoring Component, another Government agency, or a contractor, but must be independent of the program office or the prime contractor. The OTA is selected and involved as early as possible in the program.

The OTA:

• Reviews acquisition documentation for requirements that are testable and measurable• Supports the PM and T&E IPT in preparation of the TEMP• Plans, coordinates, and conducts OT&E• Reports OT&E results to the Acquisition Decision Authority (ADA), PM, User/Sponsor, and Director OT&E

T&E

Test & Evaluation


The importance of comprehensive and coordinated T&Eplanning cannot be overstated. T&E is complex, time consuming, and critically important to the program’s success. T&E planning must capture the "who," "what," "when," "where," "why," and "how" much it will cost to ensure the system meets the user’s requirements. T&E planning should be done by an Integrated Product/Project Team (IPT) or working group.

T&E planning typically includes:

• Determining types and quantity of data to be collected

• Establishing safe test procedures

• Ensuring environmental protections are in place

• Projecting resources and schedule requirements

• Integrating T&E activities with the program schedule

The key product of T&E planning is the TEMP.

Test & Evaluation Planning


page 7 of 22

T&E

Test & Evaluation

TEMP

Test & Evaluation Master Plan


The Test & Evaluation Master Plan

The program office captures all of its planning for T&E in the Test & Evaluation Master Plan (TEMP).

Select the TEMP tab to learn more about it.

D


page 8 of 22

Long Description

Interactive three ring binder with selectable tabs.

T&E

Test & Evaluation

Test & Evaluation Master Plan Purpose The Test & Evaluation Master Plan (TEMP) is a living, high-level planning document for all T&E-related activities that describes the program office's Developmental Test & Evaluation (DT&E) and Operational Test & Evaluation (OT&E) strategy; defines test parameters, attributes, and methods; summarizes T&E resource requirements; and allocates T&E responsibilities. Prepared By The TEMP is prepared by the Operational Test Agent (OTA) and Test & Evaluation IPT with oversight from the PM. Appendix DHS Instruction/Guidebook 102-01-001, Appendix L contains guidance and a sample template for the TEMP. Approval The TEMP is endorsed by the OTA and approved by the Director of Operational Test & Evaluation (DOT&E) at or prior to ADE-2B.

SELC Tailoring Plan Purpose The purpose of the SELC Tailoring Plan is to inform DHS leadership about how the SELC will be adapted to fit the needs of a specific acquisition project. The plan identifies the major activities and reviews that will be conducted, as well as the technical management artifacts that will be prepared for the project. Prepared By The Program Manager (PM) is responsible for preparing the SELC Tailoring Plan in accordance with the guidance outlined by the Program Accountability and Risk Management (PARM). Guidance The DHS Systems Engineering Life Cycle (SELC) Tailoring Plan Guide, prepared by the PARM and the Office of the Chief Information Officer (CIO), provides guidance and a sample template. Approval The Component Acquisition Executive (CAE) or the Acquisition Decision Authority (ADA) approves the SELC Tailoring Plan prior to ADE-2B.

Acquisition Plan Purpose The AP answers the question, "How can we strategically acquire this capability?" The AP contains the overarching acquisition strategy, and is used to inform and integrate activities across the acquisition, including acquisition planning and integrated support planning. The AP should provide: A clear statement of the desired acquisition outcomes A comprehensive description of the business environment An assessment of the potential supplier base for desired goods and services A top-level strategy for sustainment and support Recommendations for the overall acquisition approach and types of acquisition (e.g., asset acquisitions, services acquisitions, IAAs) Prepared by The Acquisition Plan is prepared by the PM and Contracting Specialists in the PMO. Appendix

Instruction/Guidebook 102-01-001: Appendix E points users to the DHS Acquisition Planning Guide for guidance in developing an Acquisition Plan. Programs should contact the PARM for tailoring guidance of the specific sections and AP content based on the unique conditions of each acquisition. Approval The AP is a living document that spans the life of the acquisition, and will be updated and expanded over time. The PMO starts developing the AP after ADE-1, and updates it to support ADE-2A and 2B. The Chief Procurement Officer approves the AP at ADE-2A.

Acquisition Program Baseline Purpose The purpose of the APB is to define the cost, schedule, and performance parameters for the program that the PMO will be accountable for in quantitative, measurable, testable terms. The APB serves as the PMO's "contract" with the customer and DHS headquarters. The APB must: Relate to the need expressed in the MNS Track the KPPs detailed in the ORD in terms of thresholds and objectives Be consistent with the program's LCCE Align with the resources programmed in the Future Years Homeland Security Program (FYHSP) Prepared By The PMO prepares the APB. Appendix Instruction/Guidebook 102-01-001 Appendix K: Acquisition Program Baseline provides guidance on APB development, including a sample template. Approval The ADA approves the APB at ADE-2A.

Life Cycle Cost Estimate Purpose An LCCE answers the question, "How much will the system cost over its life cycle?" A program’s LCCE serves as the cost input for acquisition decisions, programming, and budget requests to Congress. The Planning, Programming, Budget and Execution (PPBE) process is the gateway for getting resources allocated to the program, and the LCCE is used much further upstream than just the budget year in the process. The LCCE must be comprehensive (i.e., include planning, research & development,

testing & evaluation, acquisition, personnel, facilities, deployment, operations, maintenance, disposal, etc.). Prepared by Acquisition Financial Managers and Cost Analysts supporting the PMO prepare the LCCE using inputs from a variety of sources, including the results of the AoA analysis and the ILSP. Appendix Instruction/Guidebook 102-01-001 Appendix I: Life Cycle Cost Estimate points acquisition professionals to the GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs (available online) for guidance on creating an LCCE. Approval The Component approves the LCCE at ADE-2A, and the PMO will continue to update it throughout the acquisition life cycle.

Integrated Logistics Support Plan Purpose The ILSP documents the overall planning undertaken to support the ILS strategy and the infrastructure that the fielded system will require in order to remain available and ready to operate once fielded. The ILSP should address cost, schedule, and performance. Prepared by The program's Life Cycle Logistics Manager prepares the ILSP with oversight from the PM, and often with the support of an IPT. Appendix Instruction/Guidebook 102-01-001 Appendix J: Supportability and Sustainment contains guidance on developing an ILSP. Approval The ILSP is approved by the ADA at ADE-2A and updated as required.

Concept of Operations Purpose The CONOPS contrasts from the user's perspective the current way the sponsoring organization operates to execute the assigned mission with future methods of operating using potential capability

solutions. The CONOPS defines the capability in greater detail than the MNS and CDP and supports development of the AoA/AA and ORD. Prepared by The User/Sponsor develops the CONOPS in parallel and interactively with the AoA/AA and ORD. Process To develop the CONOPS, the User/Sponsor:

• Begins with a baseline CONOPS that describes current operating methods • Describes the deficiencies and gaps in the "as-is" operations (referring back to the MNS and

CDP) • Gathers data to identify potential new ways to respond to future threats and conditions and

eliminate current deficiencies • Feeds those new ideas into the AoA/AA for technical and cost analysis • Updates the CONOPS as alternatives are analyzed (AoA/AA) and develops more detailed

capability needs from identified scenarios • Finalizes the CONOPS using the preferred solution(s) from the AoA/AA

Appendix Instruction/Guidebook 102-01-001 Appendix F: Concept of Operations contains CONOPS development guidance and a sample template.

Operational Requirements Document Purpose The ORD captures the User/Sponsor’s Key Performance Parameters (KPPs) and other operational requirements. Operational requirements are high-level requirements that describe the mission, objectives, and capabilities in operationally relevant terms. The requirements documented in the ORD should be traceable to the MNS and CDP. Prepared By The User/Sponsor develops the ORD in parallel and interactively with the AoA/AA and CONOPS, with support from the PMO and SMEs as required. Process To develop the ORD, the User/Sponsor:

• Uses the parameters from the AoA/AA and CONOPS analyses to populate the ORD (the final ORD should reflect the parameters of the preferred alternative)

• Identifies and documents Critical Operational Issues (COIs) that describe what the capability must be able to do in its operational environment to meet the mission need

• Identifies and documents KPPs that must be satisfied by the capability. The minimal acceptable level for each KPP (threshold) and the maximum desired level (objective) should be provided

Appendix Instruction/Guidebook 102-01-001 Appendix H: Operational Requirements Document contains ORD development guidance and a sample template.

Approval The ADA approves the ORD at ADE-2A.

Capability Development Plan Purpose The purpose of the Capability Development Plan (CDP) is to serve as the agreement between the Component Head, the Program/Project Manager (PM), and the Acquisition Decision Authority (ADA) on the activities, cost, schedule, and performance boundaries of the work to be performed in the Analyze/Select phase. This critical period leading up to ADE-2A develops the knowledge used by the ADA to make informed decisions on the performance, schedule, and cost of the program that will effectively deliver capabilities to users. Prepared by The PM prepares the CDP with input from the User/Sponsor, cost analysts, logisticians, and other subject matter experts (SMEs). Appendix Instruction/Guidebook 102-01-001 Appendix D: Capability Development Plan contains guidance and a sample CDP template. Approval The ADA approves the CDP at ADE-1.

Mission Needs Statement Purpose The MNS describes a specific functional capability required by users to accomplish a Component mission or objective. The MNS:

• Describes the problem, NOT the solution • Is a formal description of a strategic need for investment • Initiates acquisition program management process • Is the basis for submitting a Resource Allocation Plan (RAP) to request funding

Prepared by The MNS is prepared by the User/Sponsor that identifies the need. Appendix

Instruction/Guidebook 102-01-001 Appendix C: Mission Needs Statement contains guidance and a sample MNS template. Approval The MNS is approved by the ADA at ADE-1. Example For example, a MNS identifying a gap in Customs and Border Protection’s capability for ensuring the security of the US-Mexico border might describe a need to enhance agents’ ability to detect people entering the country illegally.

Preliminary Missions Need Statement Purpose The purpose of the P-MNS is to state the need at a top-level, clearly and accurately, as a basis for developing a Mission Needs Statement (MNS) and submitting a Resource Allocation Plan (RAP) to request funding for a future acquisition. Prepared by The P-MNS is prepared by the User/Sponsor that identifies the need. Appendix The P-MNS is drafted using the same format as for a MNS, which is located in Instruction/Guidebook 102-01-001 Appendix C: Mission Needs Statement. The User/Sponsor only completes those sections for which knowledge is available. Approval The P-MNS is approved by the Component at ADE-0 and sent to the ADA for review. Additional Information The P-MNS is not a formal requirement; if the Component desires, it may develop a MNS directly without developing a P-MNS first.

Skip Navigation--> Glossary Acronyms Resources Plugins He

Knowledge Check

For each definition, select the appropriate term from the drop-down list provided. When you are finished, select Submit.

The principle tool used to measure progress and success during system development which provides vital support for acquisition decisions. T&E

The document that describes the program office's T&E strategy, resource requirements, and roles and responsibilities.TEMP

The activity accomplished by the T&E IPT which captures who, what, when, where, why, and how much it will cost to ensure the system meets the user's requirements. T&E Planning

One of the main reasons acquisition program offices conduct Test & Evaluation during the Acquisition Life Cycle.Risk Management

The role responsible for planning, coordinating, and conducting Operational Test & Evaluation (OT&E).OTA

Show my answer


page 9 of 22

T&E is the principle tool used to measure progress and success during system development. It provides vital support for acquisition decisions. The TEMP is the document that describes the program office's T&E strategy, resource requirements, and roles and responsibilities. T&E Planning is the activity accomplished by the T&E IPT. It captures who, what, when, where, why, and how much it will cost to ensure the system meets the user's requirements. Risk Management is one of the main reasons acquisition program offices conduct T&E during the Acquisition Life Cycle. The OTA is the role responsible for planning, coordinating, and conducting OT&E.


Recap: Introduction

Review the key concepts introduced in this topic:

• Test & Evaluation (T&E) is the principle tool used to measure progress and success during systemdevelopment which provides vital support for acquisition decisions.

• One of the main reasons for T&E is to reduce program risk.

• An Operational Test Agent (OTA), identified as early as possible in the program, is responsible for planning,coordinating, and conducting Operational Test & Evaluation (OT&E).

• The program office usually forms a T&E Integrated Product/Project Team (IPT) to oversee T&E planningand execution.

• The key product of T&E planning is the Test & Evaluation Master Plan (TEMP), an acquisition documentthat describes the program office's T&E strategy, resource requirements, and roles and responsibilities.

The next topic is Types of Test & Evaluation.


page 10 of 22


Introduction to Types of Test & Evaluation

A program office is primarily concerned with two types of T&E:

1. Developmental Test & Evaluation (DT&E)

2. Operational Test & Evaluation (OT&E)

Additionally, the program office may conduct one or more operational assessments (OAs) during the program life cycle.


page 11 of 22


Developmental Test & Evaluation (DT&E) is conducted throughout the engineering, design, and development process to assess progress towards meeting technical performance requirements. DT&E answers the question, "Did we build it right?" DT&E is performed by the contractor in a laboratory or test facility using components, assemblies, subsystems, and full scale prototypes.

The objectives of DT&E are:

• To assess the validity of the Analysis of Alternatives(AoA)

• To identify potential cost versus performancetrade-offs

• To identify design risks

• To identify the capabilities and limitations of asystem

• To support information systems security andstandards conformance certifications

• To determine whether or not the system is readyfor Operational Test & Evaluation (OT&E)

Developmental Test & Evaluation


page 12 of 22


An Operational Assessments (OA) is an in-progress (during development) assessment of progress toward meeting the program's Key Performance Parameters (KPP) and assessment of the system's risk of operational effectiveness and suitability at ADE 3. The OA is conducted by an Operational Test Agent (OTA) using non-production test articles (prototypes, models, etc.), typically to gauge the system’s ability to support the OT&E requirements.

An operational assessment is one component of a program’s overall test strategy; not a substitute for OT&E. Multiple OAs may be conducted at any time in the program’s life cycle using technology demonstrators, mock-ups, engineering development models, simulations, or production representative articles.

Operational Assessments


page 13 of 22

OT&E

Operational Test & Evaluation



OT&E is a series of independent field tests conducted with operational users; with full-scale, production-representative systems; under actual conditions; and in a realistic environment. Rather than using members of the program office, an independent OTA plans, coordinates, and conducts OT&E to guarantee the integrity of the results.

The fundamental purpose of OT&E in an acquisition program is to validate that the system meets the user’s operational performance requirements, or key performance parameters (KPPs), and is operationally effective and suitable.

Select each tab for additional information.


page 14 of 22

Operational Suitability

Operational effectiveness is the ability of a system to accomplish a mission operated by representative users in the planned operational environment. Can the system perform its intended tasks?

The CHEM-Bot is being acquired to detect airborne contaminants in U.S. seaports. If the CHEM-Bot's sensor is unable to accurately distinguish specified chemicals from other contaminants in the air in cargo holds of ships entering the ports, then the system will not be considered operationally effective.

Operational Effectiveness

OT&E


OTA

Operational Test Agent

Operational Effectiveness

Operational effectiveness is the ability of a system to accomplish a mission operated by representative users in the planned operational environment. Can the system perform its intended tasks?

The CHEM-Bot is being acquired to detect airborne contaminants in U.S. seaports. If the CHEM-Bot's sensor is unable to accurately distinguish specified chemicals from other contaminants in the air in cargo holds of ships entering the ports, then the system will not be considered operationally effective.

Operational Suitability

Operational suitability is the degree to which a system is deployable and sustainable in its operational environment, and takes into account reliability, availability, maintainability, interoperability, training, cyber security and safety.

The CHEM-Bot's intended operational environment is a U.S. seaport. If the robot's components are susceptible to the corrosive effects of sea water, then the system will not be considered operationally suitable.


The OT&E Process

At a high level, the process for planning and conducting OT&E includes the following steps:

1. Reviewing the Critical Operational Issues (COIs) and KPPs the system will be measured against in theOperational Requirements Document (ORD).

2. Define corresponding measures of effectiveness and suitability (ORD, TEMP).

3. Obtain production or production-representative test systems, such as Low Rate Initial Production (LRIP)units.

4. Organize typical or representative end users to operate the system during the tests.

5. Coordinate testing of the system under realistic operational conditions (OT&E).

6. Gather and analyze test data; evaluate the results and report the conclusions (OT&E Reports).

The products of OT&E are a key input for the Acquisition Decision Authority's (ADA’s) decision at ADE-3.


page 15 of 22

OT&E


TEMP

Test & Evaluation Master Plan

ADE-3

Acquisition Decision Event 3


Knowledge Check

Match the terms below with their definitions.

Activity conducted to provide timely, accurate information to stakeholders to support acquisition decisions in a manner that reduces program risk.Test & Evaluation (T&E)

Testing conducted throughout the engineering, design, and development processes to assesses progress toward meeting technical performance requirements.Developmental Test & Evaluation (DT&E)

Testing that uses full-scale, production-representative systems, operated under actual conditions by real users, to validate design against user requirements.Operational Test & Evaluation (OT&E)

An early evaluation of operational effectiveness and suitability conducted by an independent operational test agent that can be conducted at any time during the development life cycle.Operational Assessment (OA)

Show my answer


page 16 of 22

• Activity conducted to provide timely, accurate information to stakeholders to support acquisition decisions in a manner that reduces program risk is Test & Evaluation (T&E).• Testing conducted throughout the engineering, design, and development processes to assesses progress toward meeting technical performance requirements is Developmental Test & Evaluation (DT&E).• Testing that uses full-scale, production-representative systems, operated under actual conditions by real users, to validate design against user requirements is Operational Test & Evaluation (OT&E).• An early evaluation of operational effectiveness and suitability conducted by an independent operational test agent that can be conducted at any time during the development life cycle is an Operational Assessment (OA).


Recap: Types of Test & Evaluation

Review the key concepts introduced in this topic:

• Developmental Test & Evaluation (DT&E) is conducted throughout the engineering, design, and development processes to assess progress towards meeting technical specification and performance requirements.

• An Operational Assessment (OA) is an in-progress assessment toward meeting the program’s Key Performance Parameters (KPP) and an assessment of the system’s risk of operational effectiveness and suitability at ADE 3. It is conducted by an Operational Test Agent (OTA) using non-production test articles, typically to gauge the system’s ability to support the Operational Test & Evaluation (OT&E) requirements.

• OT&E is a series of independent field tests conducted with operational users, with production-representative systems, under actual conditions, in a realistic environment, to validate that the system meets the user’s operational performance requirements and is operationally effective and suitable.

The next topic is Test & Evaluation in the Acquisition Life Cycle Framework.


page 17 of 22


Introduction to Test & Evaluation in the Acquisition Life Cycle Framework

Test & Evaluation (T&E) activities are integrated into the phases, reviews, and decisions events of both the Acquisition Life Cycle Framework (ALF) and the Systems Engineering Life Cycle (SELC). This helps ensure the program office places the proper emphasis on T&E as vital acquisition function and the program receives the risk reduction benefit T&E offers.

You'll be introduced to the following concepts in this topic:

• How T&E activities align with the ALF

• How T&E fits into the systems engineering process


page 18 of 22


Test & Evaluation Across the Phases of the Acquisition Life Cycle

T&E planning and execution are integrated throughout the ALF.

Select each element of the diagram below to learn more.

D

Select this link for a summary of the information associated with this diagram: Diagram Pop-up Text Summary.


page 19 of 22

Long Description

This diagram depicts the Acquisition Life Cycle Framework (ALF) and the major T&E events in a typical program. The four phases (Need, Analyze/Select, Obtain, Produce/Deploy/Support) and six acquisition decision events of the ALF are across the top of the diagram. Below the ALF, the following T&E events are shown from left to right (in roughly chronological order). Below the Analyze/Select (A/S) Phase is a box representing ORD development labeled "ORD." Below the A/S Phase is another box representing the formation of the T&E working group, labeled "T&E IPT." Below the Obtain Phase are ten boxes, starting with the Identification of the OTA and development of the TEMP, then DT&E, broken down into planning, execution, and reporting. After DT&E comes the OTRR in the SELC, followed by OT&E, again broken down into planning, execution, and reporting. Finally, the Director, OT&E assesses the OT&E results.

ADE-0

At ADE-0, the user hasn\'t yet considered test & evaluation requirements.

ADE-1

After ADE-1, the program office works with the User/Sponsor to 1) explore what is possible based on the available alternatives and 2) document the operational requirements and concept of operations for the system.

ADE-2A

If the ADA approves the program at ADE-2A, the T&E IPT begins developing the TEMP, if they haven\'t already done so.

ADE-2B

At ADE-2B, the ADA verifies the TEMP is complete and approved to the DOT&E.

ADE-2C

At ADE-2C, the ADA will take DT&E and OA reports and overall T&E progress into account when deciding whether to authorize LRIP.

ADE-3

At ADE-3, the DOT&E will verify, through a Letter of Assessment, that the system has been adequately tested in a relevant environment and deemed operationally effective and suitable based on its performance against the user\'s requirements to support the ADA\'s decision whether to authorize full rate production.

The Need Phase

No test & evaluation activities take place during the Need Phase.

The Analyze/Select Phase

During the Analyze/Select Phase, the User/Sponsor documents the critical operational issues (COIs), key performance parameters (KPPs), and operational concepts that will become the basis for testing & evaluating the system.

The Obtain Phase

The bulk of test & evaluation takes place during the Obtain Phase, including Developmental Test & Evaluation (DT&E) and Operational Test & Evaluation (OT&E).

The Produce/Deploy/Support Phase

During the Produce/Deploy/Support Phase, field operators may conduct (optional) Follow-on Operational Test & Evaluation (FOT&E) under OTA supervision using fielded production units to 1) isolate the root causes of issues, 2)validate operational effectiveness and suitability, or 3) drive continuous improvement.

ORD

During the Analyze/Select Phase, the user develops the ORD which defines the COIs and KPPs that T&E will measure the system against.

T&E IPT

The program office forms a T&E IPT, also called a T&E working group, to oversee T&E planning and execution.

TEMP

The T&E IPT will create the TEMP to document and communicate the program office\'s test strategy and schedule, and satisfy the requirements of ADE-2B.

Identify OTA

The program manager will identify an OTA as early as possible in the life cycle to take the lead on OT&E planning and coordination.

DT&E Plans

With support and oversight from the program office, the developer will plan (and document the planning for) DT&E.

Developmental Test & Evaluation (DT&E)

The developer will iteratively test the prototypes and development models of the subsystems and final integrated system in a laboratory or other controlled environment.

DT&E Reports

The developer will document and report the results of DT&E, which factor heavily into the ADE-2C decision whether or not to authorize Low Rate Initial Production (LRIP).

OTRR

As part of the SELC, which is aligned with and complements the acquisition life cycle, the program office will undergo an Operational Test Readiness Review (OTRR) prior to executing OT&E.

OT&E Plans

The OTA plans and coordinates OT&E. The OTA documents the OT&E planning and scheduling, and works closelywith the program office to align OT&E with the program’s schedule and budget.

Operational Test & Evaluation (OT&E)

Under the direction and oversight of the OTA, the system will be tested with end users in the operational environment.

OT&E Report

The OTA will report the results of OT&E, which factor heavily into the ADE-3 decision whether or not to authorize Full Rate Production (FRP).

DOT&E Assessment

The DOT&E will review the DT&E and OT&E reports and assess the results of the program\'s T&E effort.

T&E

Test & Evaluation

ALF

Acquisition Life Cycle Framework


Test & Evaluation in the Systems Engineering Process

System developers use the systems engineering process to design, develop, and deliver our system. Let's examine how T&E fits into the systems engineering process.

D

T&E verifies that system developers produced a system that meets the user's requirements. In the systems engineering process, this is called the Verification Loop. This step ensures components and subsystems perform to their allocated requirements, and that the system performs up to the user's KPPs.


page 20 of 22

Long Description

This diagram depicts the three step systems engineering process. The inputs to the process are requirements and constraints, represented by an arrow pointing to the right into step one. The three steps are boxes labeled Requirements Analysis, Functional Analysis, and Design Synthesis. Between Step 2, Functional Analysis, and Step 1, Requirements Analysis, is the Requirements Loop. Between Step 3, Design Synthesis, and Step 2, Functional Analysis, is the Design Loop. Between Step 3, Design Synthesis, and Step 1, Requirements Analysis, is the Verification Loop, also known as Test & Evaluation. Technical Management Processes/Tools are used to manage, measure, and guide each step. The outputs of the SE Process are the specifications, drawings, and technical data the design comprises.

T&E

Test & Evaluation

KPPs

Key Performance Parameters

Skip Navigation--> Glossary Acronyms Resources Plugins He

Knowledge Check

For each definition, select the appropriate term from the drop-down list provided. When you are finished, select Submit.

When it's verified that the Test & Evaluation Master Plan (TEMP) is complete, endorsed by the Operational Test Agent (OTA), and approved by the Director of Operational Test & Evaluation (DOT&E) ADE-2B

When the Developmental Test & Evaluation (DT&E) reports and overall test & evaluation progress are taken into account in the decision whether to authorize Low Rate Initial Production (LRIP)ADE-2C

When it's verified that the system has been adequately tested in a relevant environment and deemed operationally effective and suitable based on its testing performance against the user's requirementsADE-3

When in the Systems Engineering Life Cycle (SELC) the system is judged ready to begin Operational Test & Evaluation (OT&E)OTRR

When in the Systems Engineering Process it's verified that system developers produced a system that meets the user's requirements, i.e., (the components and subsystems satisfy their allocated requirements)Verification Loop

Show my answer


page 21 of 22

At ADE-2B, it is verified that the TEMP is complete, endorsed by the OTA, and approved by the DOT&E. At ADE-2C the DT&E reports and overall test & evaluation progress are taken into account in the decision whether to authorize LRIP. At ADE-3, it is verified that the system has been adequately tested in a relevant environment and deemed operationally effective and suitable based on its testing performance against the user's requirements. In the SELC, during the OTRR the system is judged ready to begin OT&E. In the systems engineering process, during the Verification Loop, it is verified that system developers produced a system that meets the user's requirements. In other words, the components and subsystems satisfy their allocated requirements.


Lesson Summary

Review the key concepts introduced in this lesson.

T&E Overview

Test & Evaluation (T&E) is the principle tool used to measure progress and success during system development. Acquisition programs are required to include T&E to reduce risk and provide vital support for acquisition decisions. A T&E Integrated Product/Project Team (IPT) usually oversees T&E planning and execution. The key product of T&E planning is the Test & Evaluation Master Plan (TEMP), an acquisition document that describes the program's T&E strategy, resource requirements, and roles and responsibilities.

Types of T&E

A program office is primarily concerned with two types of T&E:

1. Developmental Test & Evaluation (DT&E) is conducted throughout the engineering, design, and development process to assess progress towards meeting technical performance requirements.

2. Operational Test & Evaluation (OT&E) is conducted in a realistic environment by operational users. These users operate production systems to validate both that they meet the user's requirements, and that they are operationally effective and suitable.

A program office may also choose to conduct one or more Operational Assessments (OAs) during development to gauge the system’s ability to support OT&E requirements.

T&E in the Acquisition Life Cycle Framework

T&E is integrated into the phases, reviews, and decisions events of both the Acquisition Life Cycle Framework (ALF) and the Systems Engineering Life Cycle (SELC). In the systems engineering process, T&E is used to verify that system developers produced a system that meets the user's requirements, a step called the Verification Loop.

You have reached the end of Test & Evaluation. To continue, select the next lesson from the Table of Contents.

To print this lesson, select print.


page 22 of 22

test & evaluation - gdit test & evaluation master plan (temp) is a living, ... operational...

Documents