I I

Measurement Assurance in Reliability Assessment

M. B. Carey; AT&T Bell Laboratories, Holmdel, NJ

G. N. Geissler, AT&T Microelectronics, Reading, PA

P. E. Montag; AT&T Microelectronics, Reading, PA

T. L. Yost; AT&T Microelectronics, Reading, PA

Keywords: process analysis, SPC. measurement uncertainty, minimum detectable shift,

ABSTRACT

Measurement assurance is a relatively new concept that has important applications in reliability. In this paper. we propose a methodology for developing measurement assurance programs (MAPS). A MAP is a quality assurance program for a measurement process. It demonst" that the uncertainty in the measurements on which reliability decisions are based are small enough to meet the needs of the reliability specialist The essential feature of a MAP is that it focuses on the whole physical measurement process, i.e. the operator. ihe ei~vhnment. the procedures, as well as the inmments. We apply the merhodology and supporting techniques developed in this paper to the certification of an electrical component used in a large system. We estimate the uncertainties in the measurements of this component and show how they are useful to the reliability analyst.

MEASUREMENT ASSURANCE PROGRAM

The concept of measurement assurance consists of applying the principles of quality control and quality assurance to physical measurements. The National Institute of Standards and Technology (NIST, previously known as National Bureau of Standards) defines a measurement assurance program (MAP) as follows (Ref. 1):

A measurement assurance program is a quality assurance program for a measurement process that quantifus the total uncertainty of the measurements (both random error and systematic conponenu qf error) with respect to national or other designated standards, and demonstrates that the totd uncertainty is smciently mall to meet the user's needs

The essential feature of a MAP is that it focuses on the whole physical measurement process. i.e. the o p e " . the environment. the procedures, as well as the instruments. To illusuate the impcetance of MAPS in reliability assessment, we describe an application of the methodology proposed in this paper to the certification testing of an electrical component, called EC, used in a large system.

AU ECs used in the large system go through a certification program where the ECs are aged during eight consecutive periods. Each aging period consists of heating the ECs at 150 de- C for 48 hours under electrical bias and is followed by a series of 17 measurements on each EC. In addition, two initial series of 17 measurements are taken on each EC. This leads to 10 series of 17 measurements for each EC. At the end of the program, all the measurements on each EC are evaluated by a Pedigree Review Commiuee with m e m b from manufacturing, engineering. and reliability. A decision either "to ship" or "not to ship" is made for each EC based on the stability over time of its parameters. in addition to OW criteria.

To evaluate the stability of these parameters, the Pedigne Review Commiaee must be able to distinguish a true shift in the measurements of a EC from a drift of the test equipment or from a non-significant shift assignable to noise in the test equipment or procedure. The inability to identify a true shift is likely to result in shipping ECs whose parameters have shifted or not shipping ECs whose parametem are stable. Both possibilities increase the costs of manufacturing the large system. To avoid these unnecessary costs, it is imperative to control the precision with which the measurements are taken. This is precisely the purpose of a MAP.

'Ihe rest of thii papex describes the methodology we developed to implement MAPS. and illustrates each step with results obtained from the study of the measurement pmcess of ECs.

METHODOLOGY TO IMPLEMENT A MAP

"he methodology is based on the recognition that physical measurements are generated by a process. Existing techniques developed in the past few years to analyze general processes are adapted to a measurement process. Three documents are of interest in this area: . PQMI, a guideline to manage and improve processes in general (Ref. 2),

.ASQC/Ml. an Amaican Standard document that specifies general

- ASQCW, an American Standard document in draft form that specifies

requirements for assuring the quality of calibration (Ref. 3).

general requirements for assuring the quality of measurements (Ref. 4).

We identified seven important steps for implementing a MAP. These steps are: 1.

2.

3.

4.

5.

6.

7.

Defme process and identify m e r : The 61% step consists of defining the measurement process. identifying its supplien and custanerr~, and making sure hat wmnc has responsibility for and authority over("owns') the whole process.

Identify critical measures: Once the process has been de6ned and overall nsponsibility assigned. critical measurcs to manage the process effectively are identified.

Establish precbioo of test instrument: The precision of the test instrument is determined by the variability of measurements taken over a short period of time. Identification of the potential sources of variability and estimation of their contribution to the o v d measurement variability is a necess~~y step in any improvement effort.

Establish control: Rocedures to assure that the measurement proctss is under statistical control are implemented. No prcduct decision should ever be based on data gathexed from an unpredictable measurement process. Estimation of the uncatainty of measurements gathered over a long penod of time (i.e. with diffuent operators, environmental conditions, ...) is necessary to establish measurement process conml.

Establish feedback from customer: The effect of measurement uncerlaintics on customers' decisions is evaluated. It is important for both the o m and the customers of the measurement process to undustand the limits of the measurement process, i.e. which conclusions can be reached given the variability of the process. and which cannot,

Document: Uptodate and accurate documtation is necessary to maintain the gains achieved by previous efforts and to further improve the process. Improve: Suggestions for further improvement of the process are developed, recorded, and implemented.

The rest of the paper applies these seven steps to the measurement process of ECs.

0149 144X/90/0000-0237$01.00 0 1990 IEEE

1990 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium 237

1

STEP 1: DEFINE PROCESS AND IDENTIFY OWNER

Figures 1 and 2 describe the measurement process of ECs. We use a topdown approach where the overall process is defined by identifying its output, customers, inputs, and supplier. A block diagram of the process is then drawn identifying the boundaries and major blocks of the process. Finally all relevant blocks are flowcharted.

FIGURE 1: DEFINE PROCESS AND IDENTIFY OWNER

OWNER: JOHN DOE

Conclusions from these charts are: - The measurement prwess is defined as producing one series of 17

measurements for each EC m e a s d by the process. Typically, a group (called a lot) of similar ECs is tested together.

- There are four major blocks (or subprocesses) in this process. One of them is repeated twice. The data entry and retrieval process is not considered in this paper.

- Calibration is done at every shift unless the change of shift takes place while a lot of ECs is being tested. This means that the test equipment may be adjusted more than once every other shift. This calibration interval is a possible subject for a quality improvement project.

- The testing of check standards is done before and after measuring a lot of ECs. A check standard is a stable, wellcharacterivd EC that is periodidy remeasured to determine whether the measurement process is in a state of statistical control. This is addressed in Step 3.

- The EC test engineer was identified as the owner of this measurement process. As such, he is responsible for keeping upadate and accurate documentation on the process, establishing and maintaining process control, resolvinglescalating process issues, improving the process. measuring and tracking improvement, and initiating process reviews.

STEP 2: IDENTIFY CRITICAL MEASURES

The following critical measure8 were selected to manage the. measurement process effectively:

- The length of time it takes to test one lot of 100 ECs. This depends on the operator and otha factors such as the amount of time allocated for temperature s t a b i t i o n . Before each EC is tested. measurements of forward voltage at 10 mA are made on each EC until these repeared measurements are within 0.05mV of each other. After adding a security margin of 15 seconds, the temperature is stable enough to measure the other parametas of the EC. A cooling my was added to hasten temperature stabilization prior to test With these characteristics. we found the mean time to pocess one lot of 100 ECs to be 3.9 hours, with a standard deviation of 1 hour.

- The percentage of check standards tests found out of control. During the one month study period, 6 percent of all measurements made on all check standards were out of control limits. This is also an opportunity for a quality improvement project

FIGURE 2: BLOCK DIAGRAM OF THE MEASUREMENT PROCESS OF ECs

t I I -

--%- -- !Z!m-r= STEP 3: ESTABLISH PRECISION OF TEST INSTRUMENT

The precision of the test instrument is determined by the variability of repeated measurements taken on the same unit under similar conditions over a short period of time by the same operator using the same procedures (repeatable conditions). We conducted an Error of Measurement study @OM) on 30 devices to estimate the variability 1 if the measurements made under repeatabk conditions. We identified some of the causes of variability. Their effect was reduced by changing the test procdures, and follow-up EOMs quantified the resulting improvement. Final resul s on the precision of the test instrument are summarized in Appendix I.

STEP 4: ES' 'ABLISH CONTROL

To establish if the measurement process was in statistical conool. we used check standards. MST defines a check standard as a stable. well-chanlcterized in-house device that is periodically remeasured to detennine whether the measurement process is in a state of statistical conuoL

Five check standards were monitored over the study period Two of these check standards had to be removed during this period: one for a broken lead, and the other for drift of its leakage current measurements. This reinforced the need for better procedures to select check standards This is yet another quality improvement opportunity. The study of the three remaining standards led to the estimation of the variability of the measurement process presented in Appendix 11. The major conclusions are:

- A faster and more efficient evaluation of the state of the process can be done by monitoring eight of the seventeen parameters measured. This conclusion was achieved by conducting a comlation study on the seventeen parameters. It confirms a correlation study done using the Error of Measurement Study data.

- A comparison between the results listed in Appendices I and I1 shows that, for all parameters but two, the uncertainty of the measurements is greatex over the long term than over the short term. This is expected as many more uncontrollable factors affect the long term results than the short term ones.

- The two parameters for which the uncertainty is greater over a short term period are leakage current at 2 and 3 Volts. This is explained by (1) all three check standards used to estimate the long-term variability had very low leakage current, but (2) some of the thirty devices measured during the EOM study had high leahge current. As the precision with which the test equipment measure leakage current is proportional to the magnitude. of the leakage current, the short-term variability turned out to be larger than the long-term variability. This situation illustrates the need to select check standards whose parameter values cover as much of the entire testing range of the system as possible and not to limit the selection of check standards to devices that have some desired properties (such as being rejmsentative of "good" devices being manufactured).

STEP 5: ESTABLISH FEEDBACK FROM CUSTOMER

The variability of the meaSurement pmcess affects our ability to detect shiits between two series of measurements. It is important for the Mgree Review Committee to know the minimum detectabk shift for each one of the

238 1990 PROCEEDINGS Annual RELlABlLlTY AND MAlNTAlNABlLITY Symposium

I

1.23% 3.58% 0.14%

O.% 0. % 0.8 O.% O.% 0.46 O.%

0.02% 0.13%

parameters. This minimum detectable shift 8- is defined such that (1) the probability of concluding erroneously that the device has shifted when, in fact, it has not is set to 5% (a = .05), and (2) the probability of concluding erroneously that the device has not shifted when, in fact, it has shifted by an amount equal to 8, is set to 5% ( p = .05). Details of the calculations are provided in Appendix III. Minimum detectable shift for the critical parameters primarily investigated by the Pedigree Review Committee are also listed in Appendix III. Major conclusions from these results are: - The minimum detectable shift in breakdown voltage is low (3 mV).

- The minimum detectable shift in forward voltage at 10 Amps is 13 mV. This includes the latest improvements made during the study to measure forward voltage.

- The minimum detectable shifts in leakage current at 2 Volts and 3 Volts are high compared to the mean value of these parameters. This indicates that the resolution of our test equipment does not allow us to measure effectively parameters in the low end of the measurement scale selected (auto ranging could improve this result).

- The values of a and p should be chosen to reflect customer concerns. For example. they could be selected as a function of the costs associated with erroneous conclusions.

0.01% 0.04% 0.1

0.96 0.46 O.% 0.76 O.% 0.96 0.46

2.19% 16.20%

STEP 6: DOCUMENT

Documentation is an important step of a measurement assurance progmn as it makes available knowledge of the process accumulated by selected people. The following documentation is presently available:

- This document,

- Layout for device testing procedures and check standards testing procedures,

- Up-to-date flowcharts of the process,

- Historical records of the critical measures listed in Step 2,

- Historical records on check standards selected,

- Control charts on the check standards with their control limits.

- Historical records of ections taken when the system locks out because measurements of check standards are not within their control limits.

- Roccduns to Select C h d standards,

- Records of periodic EOMS.

STEP 17: IMPROVE

Some improvements have been noted above and are presently in the planning stage. They include:

- New procedum to test check standards. h n t l y , all check standards are measured twice before a lot of ECs is processed, and twice after the lot of ECs has been messed. The new procedure will call for check standards to be measured once before and once after any lot of ECs. This implies that the control charts of check standards will have their coml limits based on moving range. An interesting paper in this area has been written by Chimera and Tukey (Ref. 5).

- Further investigation of the critical measures identified in Step 2. Once they are recorded on a continuous basis, they could be the subject of a quality improvement project.

To further improve the precision of the measurement process, we suggest a study of the calibration subprocess. In particular, then is a concern that the variability of the measurements is increased by overcalibrating.

Another a m for further work relates to the estimation of systematic components of error (also called bias). This requires that a comparison be ma& between measurements obtained from the process studied in this paper, and measurements obtained either from a more accurate process or from the customer. It is not evident at this point what other measurement process can be used for such a comparison.

CONCLUSIONS

We have described a general merhodology to implement measurement assurance programs. These programs are critical to the success of assessing reliability based on monitoring the degradation (or lack of) of devices. They involve continuous control and improvement of processes generating physical

measurements. They also involve a dhlogue between the owner of the measurement process and the reliability analyst (or customer of that process) to determine the effect of measurement uncertainties on decisions which are based on these measurements. We have illustrated a first application of this methodology to the measurement of electrical components used in a large system. Relevant to the definition of a MAP of Section 1, this particular measurement assurance program quantifies the precision of iu measurements and relates this precision to the user's needs. Further work to relate the measurements to other designated standards, such that customer's standards or national standards. needs to be done.

APPENDIX I

The model used to estimate the precision of the test equipment is a random effects model defined as follows:

y,l = v + g + e,); i=1.30; j=1,3

where y is a parameter measured on each device, say i@2V; i is the device being measured and j is the replication of the measurement v is the mean of the measurements of parameter y that could be taken on all devices under repeatable conditions. ai is the random effect of device i on the parameter y. It is assumed to be normally distributed with mean 0 and variance of. E,/ is the error due to the inability of the test equipment to repeat measurements precisely. It is assumed to be normally disaibuted with mean 0 and variance o:. U,,, is referred to as the precision of the test equipment

We estimated U,,,, v , and of from three replications made on thhty devices for each parameter listed in the following table. These estimates are combined in the table to obtain more familiar quantities.

Observe that for all "vbr" parameters the precision 6,,, is very small compared to either o or 6:; as a result, the coefficient of variation or the % EOM are approximated by 0%.

PRECISION OF TEST INSTRUMENT

Parameter

i@2V i@3V

i@5.6V

vbr@ 1mA vbr@ 15mA vbr@75mA

vbr@ lOOmA vbr@3OOmA vbr@6OOmA vbr@ lOOOmA

vf@lA vf@lOA

Test equipment precision

6-

0.13 70.0 2.6

0.04 0.08 0.18 0.14 0.18 0.20 0.20

0.16 0.11

of variation

I

APPENDIX II The model used to estimate the variability of the measurement process is a random effects model defined as follows:

y,l = p + f3, + q,,: i=1,3; j=1,3

where y is a parameter measured on each check standard, say is@2V; i is the check standard being measured and j is the replication of the measurement. p is the mean of the measurements that could be taken on check standards over a long period of time. p, is the random effect of check smdad i on the parameter y. It is assumed to be normally distributed with mean 0 and variance ob. q,) is the emr due to changes in the measurement process. It is assumed to be normally distributed with mean 0 and variance o;. up is referred to as the variability of the measurement process.

We estimated U,,, , and ui for each parameter listed in the following table from all the replicahons made on lhree check standards during the study period.

1990 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium 239

1

VARIABILITY OF MEASUREMENT PROCESS

Parameter

i@2V i@3V

W5.6V

vbr@lmA vb@ l5mA vb@75mA vbr@ loomA vbr@300mA vbr@600mA vbr@ lOOOmA

vf@lA vf@ 1OA

Measurement process d-bility

QP

0.05 0.6 4.6

0.6 0.7 0.7 0.6 0.6 0.6 0.6

0.5 2.5

Coefficient of variance

6, I c

20.3% 76.1% 0.25%

0.010% 0.010% 0.010% 0.009% 0.009% 0.009% 0.009%

O.m% 0.29%

APPENDM III I. Minimum detectable degradation when there is no replication of measuremenu

Let y1 and y2 be two measurements made at two different times on the same device. We assume that the variability of the measurement process is up, i.e. ( yi )ia,z follows a normal distribution with mean and standard deviation a,,.

We define the minimum detectable d e g " , 8- as follows. In the following test of hypothcsts:

Ho : CL^ - p1 = 0 versw Ha : k - p1 = &, 15- is defined such that the risk of type I e m . a, is set to 5% and the risk of

The decision d e associated with the above test of hypothess is based on tk statistic y2 - y1 which f ows a Normal distribution with mean - pI , type II e m , b. ki alS0 Set to 5%.

and standard deviarion up .pu 2 . M m precisely. we will:

Reject H o when I y Z - y 1 1 > z , 2 a P G . (A.1)

where zW2 is determined from the Standard Normal variable Z as:

P( I Z I > ~ , 2 ) = a

For an a of 5%. zW2 is wual to 1.96. To obtain &, we evaluate B. !3 = P ( acccpring H o I Ha is f rw )

= P ( IY2-YlI < z , 2 , 6 I k-cll=L)

= ~ ~ - ~ , 2 ~ p G ~ Y z - Y l ~ z w z u p ~ I p z - p 1 = L )

Setting p to 5% leads approximately to:

AS Z ~ Z = 1.96, and zp = 1.65, this leads to:

1 6- I = 0, G(3.61) 64.2)

Formula (A.2) was used to calculate I 8- I in the table pwidcd in this Appendix. II. Minimum k t a b k degradation when mcasurcments arc replicated

If the meaSurCments are replicated n times the first time a device is measured, and n times the second time the same device is measured, then we would USE the statistic j$ - 7 2 to test the hypotheses listed in (A.1). where YI is the average of the n meaSunmentS made. the first time, and yz is the average of the n measmcnts made the second time. -y2 follow a distribution with mean pi - & ,and with standard deviation a P & / f % same approach as the one used when no replications w u c made (*1), leads to:

240 1990 PROCEEDINGS Annual

MlNIMUM DETECTABLE S H P T &

W 2 V is@3v

i@5.6V

vbr@lmA vbr@15mA vbr@75mA vbr@ 1 00mA vb@300mA vbr@MX)mA vbr@ lOOOmA

vf@ 1A vf@ 10A

L

Average value ova measurements on 3

check s ~ d a r d s X

0.26 0.80 1.86

6.95 6.98 7.00 7.00 7.02 7.05 7.06

0.77 0.87

Minimum detectable

degradation I L l

0.27 3.0

0.02

0.003 0.003 0.003 0.003 0.003 0.003 0.003

0.003 0.013

REFERENCES

1. NBS Special Publication 676-II (1984). "Measurement Assurenct Programs: Parts 1 and 11"

2. E. Lega, R. Ackaman, R. Coleman, and J. MacDorman (1987). "F" Quality Management and Improvement Guidelines." ATBT's Customa Informarion Center. Indianapolis, IN.

3. ASQC/Ml (1987) "American National Standard for Calibration Systems."

4. ASQC/Q4 (draft) "American National Standard for Quality Control of Measurements."

5. J. L. Ciminera, J. W. Tukey (1989). "Control-Charting Automated Laboratory I n s m e n u when many Successive Differences may be Zero." J o d of Quality Technology, Vol. 21. No.1.

BIOGRAPHIES

Michue Bodanger Carey AT&T Bell laboratories Crawfords Comer road Holmdcl. NJ 07738

Michele Carey is a Manber of Technical Staff in the Reliability Rescerch and Technology Gdup of thc Quality Assurance Center at AT&T Bell lalxrratories. She is primarily responsible for developing uchniqucs to evaluate component reliability based on degradation data. She is a consultant in this am for ATdT as well as in measunmcnt assurance and analysis of failure times, She is also involved in developing and teaching workshops in reliability data analysis. Michele holds a MS in Statistics from the University of FWs. France, and a PM in Statisrics from the University of Rhode Island.

G. N. Geisslu AT&T MicroelecmNcs Reading, PA

Ge~rgc Geissler is a Seniar Quality Engineer assigned to Undersea IC Assembly and Testing at Reading. PA. He has established quality control proceduns in these rntas with thc aim of reducing variability as well as cost while improving reliability. Gaxge holds an MS in marhematics from New Jersey Imtitute of Technology, an MBA in management from Fairleigh Dickinson Univcrsity. and is an ASQC certified Quality Engineer.

RELlABILITY AND MAINTAINABILITY Symposium

I I

P. E. Montag AT&T Microelectronics Reading, PA

Phillip Montag is a product Enginta for Undmvatu Lightwave System Componeno. He is also responsible for the testing and &cation of system components Philip holds a B. S. Degree in Computa Science from lubright

Lincoln Technical Institute. Coll~?@ and an Associate Dew in Sptcialized EkXtWiC TCChnObgy from

T. L. Yost ATBT Microelecaonics Reading. PA

Teny Yosf is M Engineering Associate assigned to UndcRes Component Testing and CcrtiEcation at Reading. PA. He is primarily ruponsible f a the reliability aging and testing of subcable integrated circuirs. Tary holds 811 Associate Degree in Electrical Engineaing Technology from Pennsylvania Sate University.

1990 PROCEEDINGS Annual RELIABILITY AND MAINTAINABILITY Symposium 241

1