acquisition insight days science of test of test2.pdf · air- to -ground munitions zhdwkhu w...

Acquisition Insight DaysScience of Test

Brent Russell, ctr.SOT Course Director

AFIT School of Systems and Logistics

Scientia Prudentia et Valor

STAT T&E COE: Scientia Prudentia et Valor

What is STAT?

• Scientific Test and Analysis Techniques (STAT) are the scientific and statistical methods and processes used to enable the development of efficient, rigorous test strategies that will yield defensible results.

• STAT encompasses such techniques as • Design of experiments (DOE),

• Data regression,

• Basic statistical analysis,

• Reliability growth,

• Survey design,

• Combinatorics

2DOE is the workhorse of STAT


What is Design of Experiments (DOE)?

• Sequential, progressive test method

• Supports comprehensive analysis; main effects, interactions, and empirical modeling

• Based on proven statistical techniques and mathematical models

• Designed, randomized experiments establish correlation and causation

• Mathematically quantifiable as fewest points required for a given level of confidence

• Experimental noise is considered in the design and estimated through the analysis.

• Deals with test constraints while minimizing variance, maintaining statistical integrity.

• Empirical modeling of responses allows precise estimation of future responses.

3

“Scientific research is a process of guided learning. The object of statistical methods is to make that process as efficient as possible.” (Box, Hunter, Hunter 1978)


DOE in Defense

INPUTS

(Factors)

OUTPUTS

(Responses)

PROCESS:

Air-to-Ground

Munitions

weather, training, TLE, launch conditions

Noise

Altitude

Weapon type

Impact Velocity

Delivery Mode

Impact Angle Delta

Impact AngleImpact Velocity Delta

Miss Distance

4


5


Requirements with Ivy Hooks

• Day in the life (“Operational Concepts”)

– NOT design concepts

• “The system shall…”

• Necessary, verifiable, unambiguous

6


7

Mismatch of Requirementsand Evaluation

• Evaluation of systems against specific requirements versus performance across the operational envelope– Often requirements are narrowly-focused, don’t cover the envelope

– Static in time and do not keep pace with evolving threat

– Test scope is often limited to thesystem under test while the systemwill be operated as asystem-of-systems in a jointenvironment

– DOT&E EmphasisD

iffic

ulty

of t

he E

nviro

nmen

t

Difficulty of the Target

Tests designed to requirements alonecould limit examination of system performance

Operational Envelope

Requirements Definition


STAT-Infused Requirements

8


Other STAT Considerations

• What accuracy is required and is it the same across all conditions?

• What conditions matter?

• What if conditions change?

• How should the failing X% be evaluated?

• Are there critical conditions where failing cannot be allowed?

• How is failing performance evaluated?

• How rapidly does performance degrade once it goes below threshold?

• What analysis is required for the decision-maker?– Must the system demonstrate passing performance across the whole operational space?

– How will the space outside the specified region be evaluated?

– Is performance aggregated into a single performance distribution (across all conditions)?

– How can demos be proven with some level of assurance if they are one-off events?

– Which points will inform the average?

9


Hypothesis Testing

10

CONFIDENCE

POWER


DoDI 5000.02 Acquisition

“use scientific test and analysis techniques to design an effective and

efficient test program that will produce the required data to characterize system behavior across an appropriately selected set of factors and conditions” (DT&E)

“defensible statistical measures of merit (power and confidence)

associated with the coverage of the factors in a quantification of test risk. Specifically, the TEMP must discuss and display the calculations done to derive the content of testing and to develop the associated resource estimates” (OT&E)

11


Confidence

Seeing this small difference from the original mean is not

a rare event

Seeing this large difference from the original mean is a

rare event

s

12

Original mean & distribution

Next data point

s


Next data point

5% a level 5% a level

The selected confidence level will determine if the observed difference is “significant.”

In DOE, this is used to assess factor significance.


Power

b is the risk we incorrectly conclude there is no

difference

13

Power is the probability to detect an effect that is present. The signal to noise ratio (d/s) is used to size the test design.

s


Next data point &

distribution

5% a

b

Larger d and/or smaller s*will decrease b

and increase power

s*

5% a

b

d


14


Decomposition for Test Planning

Decomposition helps define DT scope and allowsassignment of mission related requirements for OT

15

Process Principles• Decompose system components

– Defines low level testable elements

– Screen factors early in DT

– Develop models in DT (not just point check)

• Decompose mission segments– Assign mission-related (derived) requirements

– Include significant DT factors

• Tailor test space for OT– Critical operational regions or DT risk areas

– OT events can verify DT model predictions


•Mission Relevant Factors List

•Predictive Model

•Operational Performance Expectations

Nu

mb

er

of

Fact

ors

Co

ntr

olle

d

Time

Venue•Open Air•Range•InstalledFactor Emphasis•Environmental•MissionOutput•Validate interfaces•Validate predictions•Assess environmentSTAT Benefit•Increasing Realism

Venue•Op Assessment•Mission Scenarios•ExercisesFactor Emphasis•Remaining factorsOutput•Evaluate Sys of Sys•Validate predictions•Assess OT readinessSTAT Benefit•Most Realism•Focused design

Collecting Data/Predicting Performance On Shared Measures Throughout

Increasing Realism

Venue•M&S•Lab•ChamberFactor Emphasis•TechnicalOutput•Validate design•Explore performance•Reduce riskSTAT Benefit•High Factor Control•Lower Cost

Extensive Prediction and

Modeling

More Realistic Testing

ValidatesPredictions

Leveraging Across DT and OT

Leverage DT for Information!

16


17


A Design in 28 Runs…

18

4 Continuous Factors

Response

All data is simulated

28 runs


Really Basic Analysis

19

If threshold was “zero”• 4/28 points < 0• 85.7% “passed”• But what contributed to failures?

But I am interested in much more information than this!

We can plot a histogram• Fit a curve (normal happens to fit)• We compute 17.5% actually < 0• Only 82.5% “pass”


Plot The Response vs. Factors

20

How do I interpret this data?

This is only 2 of 4 factors? How do I assess all the information?

What does this spread mean?

Is this a trend?

How does this trend relate to the other one?


What I Want To Know Is…

• What factors influence the response?

• What interactions influence the response?

• I want to

– Easily plot and interpret my data

– Analyze data against my requirement

– Predict other points in the space

– Simulate more data points

21

Can DOE Improve My Analytical Output?


DOE…Significant Factors!

22

Wow! I have 10 significant effects!• Based on a confidence level (95%)• 3 Main Effects (ME)• 3 2-Factor Interactions (2FI)• 4 Quadratic Effects (Q)

ME

ME

ME

2FI

2FI2FI

Q

QQQ

I have an estimate on the magnitude of the effects

DOE analysis has created an empirical model of my data

using these terms


What Can I Do With The Empirical Model?

23

See how well it explains my data

Get an estimate of noise in the data

Simultaneously explore the response throughout the factors space

The model can be used to better estimate how much of the space “passes”… and why

Where do I fail?By how much?

Predict the response at points not tested


DOE Delivers Powerful Analysis!

• DOE goes well beyond basic analysis

• Set analysis goals… build design to meet them

• Empirical modeling provides superior insight

– Significant factors: ME, 2FI, higher order

– Easy to plot and interpret

– Permits prediction in the factor space

– Assess performance and causal conditions

– Simulate points for additional data

24

Starting at the end shows you the benefits of DOE before you learn the “whys”

acquisition insight days science of test of test2.pdf · air- to -ground munitions zhdwkhu w...

Documents