The Statistical Analysis of the Impact of Adequate Testing on the Defense

Systems Development

Zaw P. TunGWU, EMSE PhD Candidate

Dr. Shahram Sarkani, Ph.D., P.E.Dr. Thomas Mazzuchi, D.Sc.

April 22, 2023

Overview Purpose Background Focus Area Data Extraction Source of Data Proposed Methodology Conclusion Future Work

Introduction Purpose

• Analyze the cost growth trend to determine if adequate testing plays an essential role in successful development of new systems in the DOD acquisition environment

Background• DOD is managing $1.68 trillion worth of acquisition programs• DOD’s acquisition costs has increased by $135 billion over the past 2 years for

98 major defense acquisition programs• GAO found that most of DOD’s portfolio total cost growth in past 2 years

occurred in the DOD’s largest programs, which are all in production– This means programs experience significant problems and changes well after the

programs and their costs should have stabilized

Focus Areas

GAO has continuously stated that in order to mitigate risks of uncovering problems in the later phases of the development cycle, knowledge based testing should be conducted at key junctures, specifically around the Engineering and Manufacturing Development (EMD) phase

Source: GAO (2011), Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP

Data Extraction

Mission Need

Statement

Between MS-B and MS-C• Appropriate TRL level (GAO knowledge point 1)• PDR (GAO knowledge point 1)• CDR (GAO knowledge point 2)

• Minimum design changes after this point• Manufacturing process should be stabilized before entering Production phase (GAO knowledge point 3)

• Should not have significant cost growth at this point

Beyond MS-C• RDT&E cost growth should be minimum

Data form SAR, GAO, and DOT&E Annual Report, etc. Data form SAR, GAO, and DOT&E Annual Report, etc.

Significant RDT&E cost growth in the Production and Development Phase is the indicator of insufficient T&E activities in earlier phases of the acquisition

Primary Source of Data Selected Acquisition Reports (SARs)

• RAND conducted several research studies based on SAR data and identified numerous “disadvantages" that an analyst must be aware of in order to maintain the validity of the data . Most prevalent problems are:o Failure of some programs to use a consistent baseline cost estimateo Exclusion of some significant elements of costo Exclusion of certain classes of major programs (e.g., special access programs)o Constantly changing preparation guidelineso Inconsistent interpretation of preparation guidelines across programso Unknown and variable funding levels for program risko Cost sharing in joint programso Reporting of effects of cost changes rather than their root causes

• Using SARs provides key “advantages”o SARs conform to a strict reporting format, providing consistency to the datao Those who create SAR reports receive annual SAR training, which adds to the consistency of

the data o SARs are presented to Congress, so the level of scrutiny that SARs receive in the review process

bolsters both the consistency and accuracy of the documents. Databases in general contain inaccuracies, but a database built from SAR data arguably withstands scrutiny better than most

Supplementary Data Sources DOT&E Annual Reports

• Current Activities• Cost data• Issues • Milestone dates• Recommendations

GAO Reports• Technology, Design, and Production Maturity• Cost Data• Milestone Data• Issues

Conceptual Model

Performance Indicators

Proposed Methodology Data Extraction

• Base year RDT&E dollars from SAR chosen for analysis, since these dollars exclude estimated inflationary effects

• This format facilitates conversion of the various base years of individual estimates into a single base year, making possible easy comparison across programs

Data Analysis • Relationship between early adequate testing and program success

– RDT&E cost growth Prior/Post MS-C Schedule Slip and Unit Cost Growth– Performance indicators Schedule Slip and Unit Cost Growth

Using SEM for Data Analysis Powerful multivariate techniques – Specialized version of

other analysis methods Advantages over Multiple Regression

• SEM programs provide overall tests of model fit and individual parameter estimate tests simultaneously

• Regression coefficients, means, and variances may be compared simultaneously, even across multiple between subjects groups

• Measurement and confirmatory factor analysis models can be used to purge errors, making estimated relationships among latent variables less contaminated by measurement error

• Ability to fit non-standard models, including flexible handling of longitudinal data, databases with auto correlated error structures (time series analysis), and databases with non-normally distributed variables and incomplete data

An Example of Cost Growth Due to Inadequate Early Testing

• Compare Baseline and Current Status(Cost and Schedule)• RDT&E Cost increase in Production Phase indicate

that more problems discovers during that phase • It is an indicator that program did not perform

adequate early testing

GAO Early Knowledge Based Testing Activities

Recently DOT&E conducted a systematic review of recent 67major acquisition programs that experienced delays• 56 programs (or 84 percent) had performance problems in

testing (either DT, OT, or both) 56 programs were categorized and cost growth were

analyzed and compared with other programs in the same category

Preliminary results shows that programs with inadequate testing in the earlier phase of the development has higher cost growth

Preliminary Results  Average % Growth nAircraft 171.65 16Helicopter 139.89 8Ship 116.94 8Missile 95.35 11Satellite 90.92 9Land Vehicle 84.82 6C3I 42.12 13

  Inadequate Testing Cost Growth Other Cost Growth

Aircraft 244.94% 114.64%

Missile 116.26% 77.92%

Preliminary data analysis shows that aircraft acquisition programs has highest average percentage of total acquisition cost growth compared to others programs.

Conclusion Knowledge based testing is essential in

detecting problems early in the acquisition phase

Program success depends on the knowledge obtained during earlier tests in the development of the system

Structural Equation Modeling (SEM) is a powerful statistical technique to analyze program success factors from the T&E perspective.

Future Work

Complete Data collection and data base construction

Refine Performance Indicators and Conceptual Model based on further research

Perform data analysis using Structural Equation Modeling (SEM) to test model and find the correlation between program success and “Performance Indicators”

Contact InformationZaw P. Tun

(407) 595-4684zph2k@gwmail.gwu.edu

Shahram Sarkani, Ph.D., P.E(888) 694-9627

sarkani@gwu.edu

Thomas A. Mazzuchi, D.Sc.(202) 994-7541

mazzu@gwu.edu

Backup

References• Arena, M., Leonard, R., Murray, S., & Younossi, O. (2006). Historical Cost Growth of Completed

Weapon System Programs. Santa Monica CA: RAND• GAO (2011). Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP• GAO (2011). WEAPONS ACQUISITION REFORM :Actions Needed to Address Systems Engineering and

Developmental Testing Challenges, GAO-11-806 • Hough, Paul G. (1992). Pitfalls in Calculating Cost Growth from Selected Acquisition Reports. Santa

Monica CA: RAND• Jarvaise, J., Drenzer, J., & Norton, D. (1996). The Defense System Cost Performance Database: Cost

Growth Analysis Using Selected Acquisition Reports. Santa Monica CA: RAND• Sipple , V. (2002). ESTIMATING ENGINEERING COST RISK USING LOGISTIC AND MULTIPLE

REGRESSION, Master Thesis, AFIT.• The office of the Director, Operational Test and Evaluation (DOT&E) (2011). FY 2011 Annual Report