annual pass failures vs known product drs j.k.orr 2015-07-14

1

Historical DataPASS Annual Failure Rate Versus Residual Product Defect Density

James K. OrrIndependent Consultant

[email protected]

Copyright 2015 By James K. Orr7/14/2015

2

Space Shuttle PASS Project• Data is presented for the Space Shuttle Primary Avionics Software

System (PASS).

• The organization that developed and maintained the PASS has existed in four companies: (1) IBM Corporation from March 10, 1973 through December 31, 1993, (2) Loral Corporation from January 1, 1994 to April 21, 1996, (3) Lockheed Martin from April 22, 1996 to July 3, 1998, and (4) United Space Alliance from July 4, 1998 until August 12, 2011. (Reference 1, page 13)

• Extensive data on release product defects and failures was collected by the project starting with STS-1 over the 30 year operational flight life of the program. (Reference 1, page 19 – 24)


3

Focus Of This Presentation

• This presentation focuses on a subset of Product DRs.– Product DRs are defined as found from Software Readiness

Review immediately prior to first flight of that software version until end of program

• Failures in this presentation are Product DRs that occur in either operation flight, in training, or testing in realistic scenarios – A Product DR found in a testing scenario with an unrealistic

number of failures that would result in loss of vehicle would not be counted as a failure


4

Annual Failure Rate Versus Product Defect Density

• The following chart (Page 7) combines data in a format that provide potentially significant insight to future flight systems.– The number of KNOWN Product DRs (Page 8)• These are Product DRs that were unknown at

the time of a specific Space Shuttle flight, but were later detected. In analysis of the Product DR, it was determined what was the earliest flight system in which that Product DR existed


5

Annual Failure Rate Versus Product Defect Density

– The number of failures are grouped by the year in which the failure occurred

– On Page 7, annual number of failures (Y-axis) are plotted versus Known Product DRs by year (X-axis) from page 8.

– Also shown is the predicted values and statistical confidence intervals using the Software Reliability Model developed uniquely for PASS project.• PASS Software Reliability Model Prediction of all Product

DRs (failures plus non-failures) per year shown on Page 9


6

Observations– Operational PASS software scenarios were executed at least

50 times on the ground for each flight flown• Commercial Software may have millions of users daily

– Key Observations:• 6 percent of existing errors were found each year as

failures in operational use of the software• The more undiscovered errors in the flight system, the

more failures are expected– For future space flight software systems, similar execution

failure rates are likely– For PASS software, Product DRs introduced prior to STS-1 (to

1981) were found as late as 2009 (Page 10)


7Copyright 2015 By James K. Orr

Late 1980’sLate 2000’s

STS-26

STS-114

Defects Removed Over Time

Example: 1988290 Latent Defects

16 Failures During Year

All PASS FailuresMost Found In Development, Testing Of New Capabilities or Training

Total of 19 Failures Occurred In Flight or Terminal Count Down Over Life Of Program

See Page 9 For More Information

7/14/2015IBMUnited Space

Alliance

8

Reference 1, Page 20 and Reference 2, Page 5


9

Reference 2, Page 11

Copyright 2015 By James K. Orr

Mean Time Between Failures in Calendar Days Between Product DRs By Year

7/14/2015

10

Reference 1, Page 23


Previously Unknown Errors Are Discovered Over Time.

As Errors Are Found, The Initial Quality Of Initial Release Is Better Known.

Current Quality Is Better Because Fewer Errors Remain Undetected.

11

References1. Christopher J. Hickey, Andrew L. Klausman, James B. Loveall, and James K. Orr, The

Legacy of Space Shuttle Flight Software, August, 2011, NASA JSC-CN-24428, JSC-CN-24683, presented at AIAA Space 2011 Conference & Exposition, 27 – 29 September, 2011, Long Beach California– Free at

• http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf– Also available through AIAA as AIAA 2011-7307

• http://arc.aiaa.org/doi/abs/10.2514/6.2011-73072. James K. Orr, Daryl Peltier, Space Shuttle Program Primary Avionics Software

System (PASS) Success Legacy – Quality & Reliability Data, Page 5, August 24, 2010, NASA JSC-CN-21317, presented at NASA-Contractors Chief Engineers Council 3-day meeting August 24-26, 2010, in Montreal, Canada– Free at

• http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100029536.pdf


http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf



http://arc.aiaa.org/doi/abs/10.2514/6.2011-7307

http://arc.aiaa.org/doi/abs/10.2514/6.2011-7307



annual pass failures vs known product drs j.k.orr 2015-07-14

Documents