On Non-linear Reliability Growth Edward Demko; Grumman Melbourne Systems Division, Melbourne

Key Words: Reliability Growth, Duane, AMSAA, Regression.

SUMMARY AND CONCLUSIONS

Test-analyze-and fix is the most commonly recognized method of improving equipment reliability. This paper focuses on the evaluation method used to measure t h e e f f e c t i v e n e s s of d e s i g n f i x e s implemented during tes t . A non- mathematical, practical approach is offered to evaluate non-linear growth as might be observed during an actual reliabil i ty growth tes t . Graphical i l lustrat ions presented show how the AMSAA and Duane least squares growth analysis techniques deal with non-linear growth data.

The reader should appreciate the importance of evaluating reliability growth near the end of test and the impact that the growth rate has on determining the success of a growth program.

This paper points out that both the Duane regression and the AMSAA model analyses are insensitive to discontinuities and/or sudden changes in growth slopes and rely heavily on early test data. The author offers a piecewise regression method to overcome the shortcomings of the Duane regression and AMSAA methods.

T h e author recognizes that the piecewise regression method described in this paper cannot be applied without some engineer ing judgement , However , the reader will find the use of piecewise regression a useful tool to examine reliability growth data and to evaluate growth model(s).

A case history approach is used to analyze both hypothetical and actual data representing a wide variety of growth curve shapes. These data are analyzed via AMSAA model, [Ref. 11 MIL-STD-189, Duane model Ref. 21 MIL-STD-1635 with least squares regression, and Duane model with piecewise regression. The results are evaluated graphically.

INTRODUCTION

The purpose of a growth test is to improve equipment reliability to meet specification. Therefore, it becomes important to evaluate the growth test in progress and determine when the test is complete.

T h e assumption that reliabil i ty growth character is t ic curves can be expressed as a straight line on log-log coordinates is the foundation of the most popular reliabil i ty growth measurement techniques. Such a line provides the slope measurement des i red f o r performing computations such as the instantaneous MTBF. AMSAA and Duane regression evaluation methods tend to overemphasize the influence of early data in evaluating growth, yielding misleading results. During the course of a typical growth test, the effort starts with debugging the obvious "gross" problems, and then enters a period of "weakness discovery", followed by a

a c t i o n p e r i o d o f implementation". This causes a condition where the observed fai lure ra te i s constantly changing as a result of problem d i s c o v e r y a n d c o r r e c t i v e a c t i o n implementation throughout the test.

To overcome the emphasis on early data and reduce the probability of obtaining misleading results, a piecewise regression method is offered. It is not a "cookbook" method for every case of non-linear growth test analysis, but it does offer an alternate technique whenever it becomes obvious that the method being used yields results too heavily influenced by the early data.

The Duane model is based on a empirical expression derived as a result of observations by J. T. Duane [Ref. 31, that the cumulative MTBF versus cumulative test time points fall on a straight line when plotted on a log-log scale. The model requires fitting a straight line through the data points, subjectively by "eye". A regression technique can be used to fit

'I c o r r e c t i v e

0149-144)(/93/$3.00 0 1993 lEEE 1993 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium 41 1

growth test data points to eliminate the subjective curve fitting operation. The author recognizes that least squares regression requires independent da ta points, and the use of cumulative data results in dependent data points, however, it does provide an opportunity to use a correlation coefficient which measures the goodness of fi t , (a measure of data smoothness). Using the least squares process, a line is fitted along the data points such that it lies on the average of all the data (average cumulative MTBF and average cumulative test time), and the slope of the line is set such that the graphical distances to the data points above and below the fitted line are equal. The best fit line seldom intercepts the last data point.

The AMSAA model assumes a non- homogeneous Poisson Process. The model uses a maximum-likelihood process to estimate a shape parameter and a Weibull intensity function to determine the scale parameter and instantaneous MTBF. The growth rate is equal to one minus the shape parameter. A best fit line is constructed using these parameters which is always coincident with the last data point. The AMSAA model does not yield a correlation coefficient goodness of fit statistic. The Cramer-von-Mises statistic is used as a goodness of fit parameter to determine the appropriateness of the model. The examples in this paper show that the use of the Cramer-von-Mises statistic is insensitive to significant changes in the growth curve slope and will accept the AMSAA model as being continuous (appropriate) when in fact there is a significant discontinuity in the data.

APPLICATION OF PIECEWISE REGRESSION

The concept proposed in this paper is simple. Least squares regression is performed on all data points (i.e. cumulative MTBF versus cumulative test hours.). The earliest MTBF data point is omitted for the next regression calculation. This data compression process is repeated again until only the three latest MTBF data points remain. Only the MTBF data point is decremented for the regression iterations, not the test hours or number of failures. With each regression i terat ion, the following parameters are computed:

The Growth Rate (slope) The correlation coefficient

The best fit first point MTBF The best fit last point MTBF The instantaneous MTBF at the end

of the test

T h e equat ions for the above computations are those used for the Duane models (Reference 1, pp. 33-35, 115).

The author uses the following ground rules to determine the optimal number of data points for the piecewise regression analysis :

1. For the number of test data points: use the maximum possible with a

Minimum of 3,

2. For the correlation coefficient: must be greater than 0.9 unless a minimum

of 3 points cannot be realized, in which case the correlation coefficient must

be greater than 0.8.

3. For the best fit MTBF last point: select the "best fit MTBF closest to last point

cumulative MTBF.

The above rules are not inviolate, they are guides. The analyst must use his or her engineering judgement in applying the above rules. Table 1 shows a computer printout of the piecewise regression using data points for sample #2. The sample contains 12 data points. The last point cumulative MTBF is 1300. The "General Statistics" in Table 1 present the cumulative MTBF best f i t points, growth rate, correlation coefficient and instantaneous MTBF for each range of data points. The first entry (all points 1 through 12) is the least squares regression results for all twelve data points. The next line shows the analysis of data points 2 through 12. Successive entries omit the earliest data point. The eleventh entry presents (points 11 through 12) a trivial solution of a best fit regression for only the last two data points. Review of these data show that for the range of points 3 through 12, the best fit for the last MTBF data point is 1286.6, the next line (points 4 through 12), shows the best fit last point to be 1321.9 which is much greater than the last point cumulative MTBF of 1300. Consequently, data points 3 through 12 were selected as candidates for the piecewise analysis. A check of the correlation

412 1993 PROCEEDINGS Annual RELlABlLITY AND MAINTAINABlLlTY Symposium

-~ ~-

coefficient finds a value of 0.997 (well above the ground rules). Therefore, data points 3 through 12 meet the above ground rules of the piecewise regression process

ANALYSIS OF RESULTS

Table 2. provides a summary of the analyses for the six sets of sample data.

Sample # l . (Linear Growth, Figures 1A & 1B) depicts hypothetical data which d e s c r i b e s c u m u l a t i v e M T B F versus cumulative test time in the form of a straight line for log-log coordinates, The Duane model using least squares and the AMSAA model yield different growth rate slopes. It should be noted that for large sample sizes of data (greater than 20), the difference is more pronounced. The slope resulting from the AMSAA model analysis is less than the slope of the straight line through the data points (see Figure lb). The author makes no attempt to explain this unexpected result.

Sample #2. (Increasing End Growth, Figure 2), depicts a growth test with little growth in the early stages of the test, followed by a high rate of growth when the effects of corrective action take hold. It is clear from Figure 2 that both the AMSAA and the regression methods do not adequately evaluate the growth during the last two thirds of the test. A linear regression of all the data points yields a growth rate of 0.623 with a correlation coefficient of 0.989 which is well within the guidelines indicated above. However, the best fit last point is 1150.5 which does not agree with the last point cumulative MTBF of 1300. Inspection of Figure 2 indicates that the regression best fit is not compatible with the MTBF performance at the end of the test. The AMSAA model results show a growth rate of 0.599 with a Cramer-von- Mises goodness of fit of 0.041 which is well within the maximum critical value of 0.214 (95% confidence). The results indicate that the AMSAA model is appropriate. This sample data also shows that both least squares regression of all the data points and the AMSAA analysis do not consider the significant growth of the MTBF after 800 test hours of test time. In other words, the data associated with the first 800 hours of test time influenced the growth measurement of the last 14800 hours.of test time and yielded

a more pessimistic growth result. The piecewise regression yields a growth of 0.704 which is consistent with the graphical representa t ion .

Sample #3 (Decreasing End Growth, Figure 3) depicts the results of a test of hardware experiencing negative growth near the end of test. The Duane least squares regression method indicates an overall positive growth rate of 0.325. The AMSAA model yields a neFative growth rate of 0.025. The piecewise regression method yields a p e e a t i v e growth rate of 0.762 for the last six points. Both the Duane regression and the AMSAA methods credit the test for growth based on early test results, when in fact the growth rate is seriously decreasing at the end of test. This condition yields results more optimistic than the piecewise regression.

Sample #4 (Decreasing, Increasing Growth, Figure 4) depicts a test with very few initial problems, then a series of failures are observed with no corrective action applied for a significant portion of the test. Subsequently, corrective actions are incorporated (perhaps as a massive upgrade) such that during the remainder of the test a deceptively high growth appears. The Duane least squares regression model yields a growth rate of 0.022, the AMSAA analysis yields a growth rate of 0.076. From inspection of the growth curve, it can be seen that the rate of growth after 300 hours is high. The piecewise regression was able to identify this strong growth for the last 800 hours of test, and yielded a growth rate of 0.471. It is another example where both the regression of all the points and AMSAA analyses are too heavily influenced by early data.

Sample #5 (Flat End Growth, Figure 5) depicts a test where growth ended about halfway through the test. Clearly, the equipment reached maturity. However, both AMSAA and Duane regression methods show growth at the end of test with growth rates of 0.213 and 0.464 respectively. The test data clearly show that reliability maturity has occurred after 4900 test hours. The piecewise regression yields a growth rate of 0.00.

Sample #6 (MIL-STD-338, Figures 6A & 6B) presents data published in MIL-STD-338. When all of the data are viewed as shown in Figure 6A, it appears that linear growth was achieved. However, if these data are

1993 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium 413

expanded from 3022 hours to 8063 hours as shown in Figure 6B, the MTBF is shown declining during the last fifth of the test time. Neither AMSAA nor Duane regression methods recognized this decline, and t h e r e f o r e y i e l d o p t i m i s t i c r e s u l t s . Regression of all the data points yields a growth of 0.535. The AMSAA method results in a growth rate of 0.361. Piecewise regression however, sensed the down trend at the end of test and yielded a growth rate of 0.499.

The above differences in measured growth rates are further amplified when the instantaneous MTBFs are reviewed in the lower half of Table 2. In some cases, the instantaneous MTBF varies by more than 40%.

Table 3 lists the data points used in all six samples.

WHY THE EMPHASIS ON THE GROWTH AT THE ENDOFTEST?

The growth slope during the final phases of a growth test in conjunction with the last data point are used to compute the "current" MTBF, commonly known as the instantaneous MTBF. The instantaneous MTBF is equal to the cumulative MTBF divided by one minus the growth rate. Therefore, in order to obtain a true estimate of the instantaneous MTBF at the end of test, the growth rate at the end of test is requi red .

The growth slope at the beginning of the tes t has l i t t le bearing on the instantaneous MTBF at the end of test. If the cumulative MTBF is decreasing at the end of the test, then the current MTBF must be less than the cumulative MTBF at that point in time (the converse is also true). Therefore, the linear portion of the growth near the end of test is of primary importance. It is used to determine the progress, end and success of a growth test.

ACKNOWLEDGEMENT3

I wish to express my appreciation to my colleagues at Grumman Melbourne Systems Division, Dr. Leonard Doyon, Billy Debusk, and Charles Contess for their assistance in preparing this paper.

REFERENCES

l."Reliability Growth Management", M I L -

2."Reliability Growth Testing", MIL-STD- 1635 3."Reliability Growth Analysis Using the Duane and AMSAA Models", &KLkh&d Brief, September, 1988 4. Meter J., Wasserman W., Kutner M. H., ADDlied Linear Statistical Mode 1s. Richard D. Irwin, Inc., Homewood, IL, 1990, pp. 370-374

HDBK- 189.

BIOGRAPHY

Edward Demko Reliability Manager Grumman Melbourne Systems Division Melbourne, Florida 32902-9650 USA

Edward Demko jo ined Grumman Melbourne Systems Division February 1988 as Manager of Reliability. He has held reliability positions from 1958 with Singer Kearfott, Lockheed Electronics Company and Picatinny Arsenal. Mr. Demko received BSME and MSEE degrees from Stevens Institute of Technology, as well as an MBA from Fairleigh Dickinson University. Mr. Demko has published four technical papers including two for the RAMS.

414 1993 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium

TABLE I. S M P L E a2 PIECEWISE REGRESSION ANALYSIS

I N W T ORTA:

TABLE 3

sAnPLE OATA

sAnPLE cun HRS cun FAILS SMPLE cun HRS cun FAILS ai 100 1 *1A 100 1

2 353 2 7-47 7 737 3 353

2 3 9 5 6 7 8 9 10 11 12

2 120 60.0 3 290 80.0 9 900 100.0 5 600 120.0 6 890 140.0 7 2100 300.0 8 9000 500.0 9 6300 700.0

. _. 1- 9 1866 5 2600 6 3990 7 9386 8 r t35 9 5680 10

12i9 9 1866 5 2600 6 3990 7 %E6 E 5935 9 6580 10 7e25 11

10 9000 900.0 11 12000 1090.3 12 15600 1300.0

Ita SEE TABLE 1 *3 30

9170 12 13 10610

12190 13760

1 i y 15 16 17 I B 19 20 i?l 22 23 2 Y 25 26 27 28 29 30

eo 150 290 350

2 3 Y 5

15'&70 17270 _ _ ~ 19160 21190 23210 25370

900

750 800 850 900 950

700 6 7 8 9 10 11 12

27610 25y10 x395 B E 3 0 37-5

1 to I2 12 1150.5 0.623 0.989 sf'&9.7 2 to I#? 11 1239.6 0.670 0.999 3937.2 3 to 12 10 1286.6 0.709 0.997 Y)388.5 9 to 13 9 1321.9 0.731 0.999 9830.0 5 to 12 8 13i5.2 0.752 0.999 SZ36.0 6 to I3 7 1355.3 0.76Z 0.999 5970.2

8 to I3 5 1311.2 0.701 1.OOO 9353.9 7 to12 6 13~8.2 0.730 0.999 9816.8

-. _ _ ~ w090 92785 95610

1 2 3 * S 6 7 8 9 10 11

YE515

*5 100 .MO 900 1600

1 2 s t o l e 9 1303.4 0.682 1.000 -8.6

10 to I2 3 1300.0 0.669 1.000 3SE2.0 11 to 12 2 1300.0 0.669 1.000 3322.0

REGRESSION LINES: ---------

3 9 5 6 7

2500 3600 900

1100 8 9 IO 7000

7700 11

1 to 12 ZE.0 1150.5 106.0 w9.7 i! to 12 97.9 1239. 6 lE1.7 3937.2 3 to 12 68.2 1286.6 -0.1 WEBB. 5 9 to I t 90.9 13s!l.9 371.5 via0 -0 - _ _ _ _ ~ ~~ ~- ~~ ~

5 to 13 116.2 1 M .2 983.3 5 U 6 . 0 6 to 12 196.1 1355.3 se9.1 5970.2 7 to 12 307.2 133.2 1111.6 ~ 1 6 . 6 to 18 m . 8 1311.2 1679.6 Y353.9 9 to l a 702.3 1303.9 2201.6 9088.6 10 to 12 900.0 1300.0 2715.2 3922.0 11 to IZ 1090.9 1300.0 xs1.2 3 2 2 . 0

TABLE 2

sunnww OF sAnPLE OATA ANALYSIS

AnsAa OUANE PIECEW I5E

GROWTH CORR. GROWTH CORR. GROWTH BOOOFJESS CRITICAL SMPLE RATE COEFF. RATE COEFF. RATE OF FIT VALUE

1 0.950 1.000 0.950 1.000 0 . Y W 0.016 0.212

0.623 0.989 0.709 0.997 0.593 0.091 0.219 3 0.325 0.371 -0.762 -0.997 -0.025 0.128 0.219 Y 0.022 O.O?l 0.971 0.972 0.076 0.095 0.21'& 5 0.969 0.995 0.000 1.000 0.213 0.067 0.219 6 0.535 0.989 -0.999 -0.813 0.361 0.199 0.218

in 0.~150 1.000 0.950 1.000 0.397 0.006 0.217

DUANE PIECEWISE MSAA

GROWTH GROWTH GROWTH SAnPLE RATE PITBFI RATE nTBFI ReTE nTEFI

1 0 . *so 0 . 950 0.623 0.325 0. 022 0 . 969 0.535

1196.3 2990. 9

0.950 0.950 0.709 -0.762 0.971 0.000

-0.999

1196.3 2 9 0 . 9 9388. 5 99.9

700.0 139.5

me. 9

0. 999 0.397 0.599 -0.028

1189.5 S E 3 . 9 3293. e 77.3

1A 2 3

3999.7 117.2 102.3

1305.3 933.5

9 5 6

0.016 0.213 0.361

. . .- 108.2 889. I 31s. 6

1993 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium 415

Figure/Sample 1A. Linear Growth Cumulative MTBF (hours)

/ ---. . . .

10 10 102 103 io4

Test Time (hours) G w t h nakz Rwss ion = 0.450, A M W = 0.444, F5- = 0.450

Figure/Sample 2. Increasing End Growth MTBF (hours)

1

102 lo5 104 lo6 10

10

Test Time (hours) Growth mlc: RegteseiOn = 0.823, AMSAA = 0.599, F5ecewise = 0.704

FigurdSample 4. Decreasing, Increasing Growth MTBF (hours)

J 10 102 lo5 lo4

Test Time (hours) G m t h nakz Repss im = 0.022, AMSAA = 0.761, Piecewise = 0.471

Figure/Sample 6A. MIL-HDBK-338 Example Cumulative MTBF (hours) lo5 1

0.1 1 10 102 io3 to4

Test Time (hours) Gmwh Rcgresbn = 0.535, AMSAA = 0.381, pieQ.Rnse . = -0.499

Figure/Sample 1 B. Linear Growth Cumulative hdlBF (hours)

lO4 7'

J 10 lo2 Id lo4 Id 10 I 4 ' ' 1 ' 1 1 ' ' " " " 1 I ' 1 ' ' ' ' ' 1 ' '

Test Time (hours)

Figure/Sample 3. Decreasing End Growth MTBF (hours)

10 10 102 io3

Test Time (hours)

Figure/Sample 5. Fiat End Growth

Gnwh R e p s i o n = 0.325, AMSAA = 4.025, PkQWise = -0.762

MTBF (hours)

I

1 0 2 io3 104 Test Time (hours)

Guwth mlc: Regression = 0.464, AMSAA = 0.212, Piecewise = 0.0al

10 10

Figure/Sample 6B. Expanded MIL-HDBK-338 Example Cumulative MTBF (hours) I

102 I J io3 lo4

Test Time (hours) Gmwth rate Piecewise = -0.499

-0- PIECEWISE AMSAA --A- REGRESSION - * OBSERVED MTBF

416 1993 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium