apollo experience report problem reporting and corrective action system

8/8/2019 Apollo Experience Report Problem Reporting and Corrective Action System

1/23

N A S A T E C H N I C A L N O T E NASA TN 0-7586

*oVI

o

e

APOLLO EXPERIENCE REPORT -PROBLEM REPORTING ANDCORRECTIVE ACTION SYSTEM

by T. J. AdamsLyndon B. Johnson Space CenterHouston, Texas 77058N A T I O N A L A E R O N A U T I C S A N D S PA CE A D M I N I S T R A T I O N W A S H I N G T O N , D . C. F E BR U AR Y 1 9 7 4


2/23

~~ _I ..1. Report No.D - 7586

7. Author(s1T. J. Adams, JSC

2. Government Accession No.

~~~ ~~~

9 . Performing Organization Name and Address

17 . Key Words (Suggested by Author(s1)' Failure Analysis' Management Information System s' Products Reliability* Spacecraft ReliabilityManagement Analysis

Lyndon B. Johnson Space CenterHouston, Texas 7705812. Sponsoring Agency Name and Address

National Aeronautics and Space AdministrationWashington, D. C. 2054615. Supplementary Notes

18 . Distribution Statement

Cat. 31

3. Recipient's Catalog No.

19 . Security Classif. (of this report) 20. Security Classif. (o f this page) 21 . NO . of PagesNone None 23

5. Report DateFebruary 1974

6. Performing Organization Code

22 . Price"$2.75

8. Performing Organization Report No.JSC S-384

10 . Work Unit No.9 1 18- 0- 0- 2

11 . Contract or Grant No.

13. Type of Report and Period CoveredTechnical Note

14 . Sponsoring Agency Code

16. AbstractA review of the Apollo spacecraft Pr oblem Reporting and Correc tive Action System is presented.The evolution from the earl y system to the pres ent day system is described. The deficienciesand the actions taken to c or re ct them a re noted, as are management control s for both the con-tra cto r and NASA. Significant experience gained from the Apollo Probl em Reporting and Cor rec-tive Action System that may be applicable to future manned spac ecraft is presented.


3/23

CONTENTS

Section PageSUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2ACKGROUND AND PRESENT SYSTEM DESCkIPTION . . . . . . . . . . . . . .Reporting of Significant Pr ob le ms . . . . . . . . . . . . . . . . . . . . . . . . 5Reporting of Routine Problems . . . . . . . . . . . . . . . . . . . . . . . . . 6Storage and Retrieval File . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Immediate Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Routine Notification and Problem Status . . . . . . . . . . . . . . . . . . . . 6Problem Closeout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Potential Hardware Impact Problems . . . . . . . . . . . . . . . . . . . . . . 11Management Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

PERFORMANCE OF THE PROBLEM REPORTING AND CORRECTIVEACTION SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16CONCLUDING REMARKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16APPENDIX -- DEFINITIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

ii i


4/23

TABLE

Table PageI SUBSYSTEM FAILURE DETECTION . . . . . . . . . . . . . . . . . . 1 5

FIGURES

Figure Page1 Early Failure Reporting and Corrective Action System . . . . . . . . 32 Pre sen t Prob lem Reporting and Corrective Action System

(a) Contractor flow . . . . . . . . . . . . . . . . . . . . . . . . . . . 5(b) Inte rnal MSC interf ace . . . . . . . . . . . . . . . . . . . . . . . 53 Example of data element matr ix fo rm . . . . . . . . . . . . . . . . . 74 Example of fai lur e investigation action re po rt . . . . . . . . . . . . . 85 Example of prob lem notification re po r t . . . . . . . . . . . . . . . . 96 Example of ch ar t showing problem sta tus to su pport an Apollomiss ion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 07 Example of problem control sheet . . . . . . . . . . . . . . . . . . . 1 28 Apollo hardware cumulative problem experience

(a) Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 3(c) Ground support equipment . . . . . . . . . . . . . . . . . . . . . 1 3(b) Spacecraft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 3

9 Apollo hardware problem experience .(a) Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 3(b) Lunar module. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 3(c) The LM ground suppo rt equipment . . . . . . . . . . . . . . . . . 1 3(d) Command and ser vi ce module . . . . . . . . . . . . . . . . . . . 1 4(e) The CSM ground suppor t equipment . . . . . . . . . . . . . . . . 1 4(g) Government furnished equipment . . . . . . . . . . . . . . . . . 14. . . . . . . . . . . . . . . . . . . . . .f) Guidance and navigation 1 4

iv


5/23

APOLL O EXPER IENCE REPORTPROBLEM REPORTING AND CORRECTIVE A CT ION SYSTEM

B y T . J. A d a m sLyndon B . J o h n s o n S pa ce C e n t e rS U M M A R Y

The Apollo spacecraft Problem Reporting and Corrective Action System was de-signed to en sur e that rapid identification and reporting w ere accomplished and that vig-oro us analys is and disposition of prob lem s were made before flight. To accomp lishthis goal, various techniques wer e used and refinements made during the pro gr am, re-sulting in the presen t closed-loop approach. Every problem was carefully analyzed,and rec urr enc e control w a s initiated to ensure ea rly m aturity of the hardware. A largenumber of open pro blem s existed in the 1965-1966 time fr am e, and many mea ns wer eused by the NASA Lyndon B. Johnson Space Center (formerly the Manned SpacecraftCenter) and the contractor s to resolve these problems. The fact that these problemswere resolved and closed in a relativ ely sho rt period of tim e is a credit to all concerned.Fe at ur es of the Pr ob lem Reporting and Corrective Action System used in the ApolloPro gra m a r e applicable to future manned spacecraft, but c ar e should be exerci sed toadapt the sy st em to the requir eme nts of the new applications.

I N T R O D U C T I O NPro ble m report ing and disposition were of significant importance in achieving

maturit y of the Apollo space cra ft hardware. Recognition of thi s importa nce w a s inade-quate in the earl y phas es of the program, resulting in the need for many refinements.The present Apollo Problem Reporting and Corrective Action System (PRACAS) is dis-cussed and background pres ented on deficiencies that existed in the e arly sy st em s andthe methods used to co rr ec t these deficiencies.

Ea rly in the Apollo P ro gr am , the following ground r ule s wer e established.1. No flight sh all be launched with unresolved or unexplained problems .2 . A l l prob lems m ust be analyzed to establish the caus e so that corrective action

can be taken o r the r i sk of not taking action can be explained.


6/23

3. The closeout criteria must include a documented cor rect ion (that is, drawingchanges, specifications, procedures, processes, and s o forth) that is applicable toeith er the hardware, software, or both.

The present sys tem was designed using these ground rule s and has the followingrequirements.

1. A l l problems occurring from acceptance tes ts through flight missions mustbe reported. Problems occurring before acceptance tests must also be reported, butfor mal management review and closeout are unnecessary unles s the problem is con-sidered critical fro m a schedule and cost standpoint.

2. An analysis of the problem is to be made by knowledgeable, design-cognizantindividuals to ascert ain t he caus e of the problem and to devise an acceptable cor rect iveand preventive action.3. The analysis and the cor rective action taken for each problem a r e to be tech-nically reviewed and veri fied independently by Reliability and Management personne l

within the contractor organization. The purpose of th is review is to confirm adequatetechnical actions and recurrence control to meet program schedules on all "like"hardware.

4. An independent review is made by Safety, Reliability, and Quality Ass urance(SR&QA) and Management pe rsonnel of the NASA Lyndon B. Johnson Space Center (JSC)(former ly the Manned Spacecraft C ente r (MSC)) to asce rt ai n the technical adequacy ofthe contractors' problem closeouts and als o the adequacy and effectiveness of thePRACAS itself.5. Continuously updated management visib ilit y of the s ta tu s of all open problems

is provided.6. The contrac tors are required to r epor t critica l pro blems promptly to NASA.

BACKGROUN D AND PRESENT SYSTEM DESCRI PT IONIn the early stag es of the Apollo Progra m, contrac tors and suppl iers used the ir

existing corporate proced ures fo r problem reporting, analysis, and correct ive action.The system primarily us ed led to the use of an electronic data pr ocessing (EDP) sys-tem as noted in figure 1. The pri mar y featu re of the early system was reporting thestat us of failures with the us e of the EDP system. The contractors e ntered the failureon computer tapes and updated the tape s at regular interva ls until the problem was re-solved. All spacec raft and ground suppor t equipment (GSE) failures were required tobe entered. Individual fail ures wer e requ ired to be closed within 30 to 45 days afterocc ur ren ce . Copies of the tapes were provided to NASA, and weekly prin tou ts weremade and distributed to cognizant MSC pers onnel. Appr opr iate NASA pers onnel re-viewed the tape printouts and provided co ncurre nce o r nonconcurrence with the con-tr ac to r' s proposed closeout. Copies of the tape printou t containing the NASA com ment swere provided the contractor for his information o r action. Printout s umma ries of thetap es w er e provided period ically to NASA management. It was never possible to obtain

2


7/23

7

Figure 1. - Early Failu re Reporting and Correcti ve Action System.

Fix formission(vehicle)

cu rr en t information utilizing the tape s. The NASA technical personnel we re able tomaintain cu rre nt information by working with their contractor counterparts.

*

Cri tic al fai lu re s were repo rted to NASA within 2 4 hour s of occurrence. Criticalfailures wer e defined as those occurring during qualification/certification test and thoseimpacting schedules, cost, o r launch (program impact failures).

Failure anal ysisl Implement Testrecommendations action iif required)- orrective action c correciive I w verification IBondedstoreroom

Thi s s yst em existe d fr om the beginning of the prog ram without significant changeuntil 1966 when tw o changes we re made. First, the requirement for reporting all GSEfa il ur es to NASA was changed to r epo rti ng only those fa il ur es of GSE us ed in countdownand of o th er GSE that had a safety o r spacecraft impact. This change was made becauseof the large number of GSE fai lur es and the need to emph asize those fai lur es re quir ingre cu rr en ce control. This requirement, however, did not relieve the contractor of theresponsibil ity fo r continuing to analyze and clos e all GSE fa ilur es . Second, the con-tr ac to rs pre par ed a n Apollo Probl em Summary (APS) of each problem for us e a t theFlight Read iness Review (FRR). The problem summ ar ie s ser ved two purposes:(1)they summarized major program impact problems for the FRR board, and (2) theyexplained problems that would not be corrected before flight.

TIsolation toreplaceable klevel

2

hardware

1

It became apparent in late 1966 and early 1967 that sev era l deficiencies existedin the syste m that required correction. Centralization of the cont racto rs' fai lur e re-porting and corrective action system w a s needed. Failu res we re not accumulated at o rmanaged from a cent ral location. This resulted in the lo ss of repo rted failures; inade-quate aware nes s of the status of prob lems by contracto r and NASA management ;

A LFix verification

ic Cor r e tP._-- Inputs tomilestone(failure,unsatisfactorycondition, A P S ,etc.)

ContractorresponsibleorganizationCorrective action feedback (reli abili ty,quality,engineering,manufacturing,etc.

.-n

A A

1 -

3


8/23

inadequate support of f ailu re closeouts to meet prog ram milestones; and poor us e ofavailable resourc es. In addition, with variou s Apollo spacecr aft being fabrica ted duringthis time frame, it w a s mandatory to have be tter cont rol of the effectiv eness of changesresultin g from failure closeout. A s a result , centralized problem asses smen t ar ea swere established. All failures were reported to these ar ea s. Representatives fromall disciplines concerned with failure closeout were assigned to these areas. Timelines were established for each failure to show vehicle and GSE closeout effectivity tosupport vehicle milestones. Thes e time line s wer e displayed to provide managementawareness.

Many problems that existed w er e significant but did not fall within the definitionof a failure and therefore wer e not reporte d o r tracked within the ove ral l sys tem. Suchi tems as dented tanks, contamination, and so forth, wer e included in this category andwer e considered unsatisfactory conditions ra th er than fail ures . The se types of prob-le ms were handled by the contract ors' interna l s ys tem but they wer e not report ed toNASA even though they we re significant. Accordingly, the name and content of theFailure Reporting and Corrective Action System was changed to Problem Reporting andCorrective Action System to encompass both failures and unsatisfactory conditions.(The PRACAS nomenclature is defined in the appendix. )

The Resident Apollo Spacec raft P ro gr am Office (RASPO) at each prime contractorplant w a s the prime interface with the contractor for problem closeouts. These re si -dent personnel wer e located in the contractor problem as se ssm ent ar e as and were re-quired to concur in all problem closeouts. This permitte d a mo re timely review ofcontractor activities and a mo re time ly NASA concurrence of closeouts . Coordinationw a s accomplished with MSC on all problems.

To improve reporting of prob lems fro m the pr im e contractor supplie rs, qualityassurance (QA) delegations'to Government agencies in residence at the supplier plantswe re amended to requ ire report ing of significant prob lem s to MSC. Thi s provided acheck on the adequacy of suppli er repor ting to the pr im e cont ract or, although, in somecas es, the response fr om the Government agency was inadequate.

In late 1967 , after repeated attempts to s treamline the EDP sys tem to make itmo re timely, the contra cto rs finally abandoned it as a real-time syst em and began re -porting all problems to MSC by datafax through the local RASPO. The open problemli st (OPL) was instituted by MSC, at fi rs t manually and then by automated printout, totr ac k open problems. The MSC RASPO w a s required to approve all contractor problemcloseouts. The EDP sys tem w a s still used, but only as a data bank for the history ofproblems. Personnel at the launch site were required to r ep ort problems in real timeto the prim e contrac tor. In addition, launch si te personnel we re given copies of theproblem closeout packages on pro blem s rep orted fr om the launch sit e and copies of theOPL. A t this time , the Apollo Configuration Change Board began to review chronicopen problems to provide management incentive for closeout. A procedure entitled"Explained Problems'' w a s added to the sy ste m and w a s presented to the FRR boardbefore each mission. The se pro blem s wer e understood but we re not closed. However,the failure mode and its effe ct wer e understood and suf ficient information w a s developedto justify the r i s k , i f any.

4


9/23

-With these improvem ents, PRACAS stabilized. The contractor portion of thesystem is shown in figur e 2(a) and the internal MSC inte rfa ces in figu re 2(b). The sy s-

tem is summarized in the following sections.

Problem analysis/ Implementrecommendations actionw correct ive act ion , c orrect ive ver i f icat ion

Bondedstoreroom- -

tI olat ion toreplaceablehardwarelevel

aLEI

T

responsible 1 I louts t0 1C=-i-n organizat ionCorrecti ve action feedback (rel iabi lity,

engineer ing,manufac tur ing ,1 tc.1Problemrepor twritten I tssessment Pr ime -area at Software contrac tor Hardcopy- ite of assessment ._3~ -Loccurrence area nApprovalldi sapproval,

Design problems MSC

Problem -ot i f i ca t ion'Iji

Contractor Hardcopy Contrac tor MSCassessment w EDP - ai lure dataarea data bank center-(b) Internal MSC interfac e.

Figure 2. - Present Problem Reportingand Corrective Action System.

R e p o r t i n g of S i g n i f i c a n t P r o b l e m sThe contractors were required to re-

port, within 24 hours of oc cu rr en ce o r de-tection, all problems that occ urred duringo r afte r acceptance testing that could ad-vers ely affect safety, contribute to scheduledelay, o r resu lt in design change; als o allproblems that occurred during certificationtesting or while settin g up equipment fo racceptance tests . If recurrence of the prob-lem on like hardw are had safety implications,the hardware supplier w a s required to pro-vide recommendations for usage restrictionsuntil problem analysis and resolution wer ecomplete. Also, the hardware supplier wasrequired to forward to MSC problem re po rt s

5


10/23

received from subti er hard ware suppliers within 24 hours of receipt. The rep ort con-tained, at a minimum, the data shown in fig ure 3. Within 1 calendar week of occur-rence o r detection, the additional data shown in the second column of fig ur e 3 wererepo rted to MSC.

R e p o r t i n g o f R o u t i n e P r o b l e m sThe contractors wer e required to report , within 1 calendar week of occurre nce o rdetection, routine probl ems that occ urr ed during o r after acceptance testing that wouldnot adversely affect the progr am. The minimum data reported a r e shown in figure 3; asam ple of one fo rm at us ed is shown in figure 4. Reporting continued f o r the operationallife of the equipment.

S t o r a g e a n d R e t r i e v a l F i l eA permanent storag e and retrie val f ile of problem s w a s maintained by MSC based

on EDP tapes submitted by the cont racto rs. This file was used for var iou s types of en-gineering investigations based on problem history.

Im m e d i a t e N o t i f i c a t i o nIncoming problem re po rt s w er e reviewed by MSC to dete rmi ne which ones shouldbe brought to the immediate attention of progra m management. A "Problem Notifica-

tion" fo rm , shown in figur e 5, was completed f o r each problem thus categorized.le m notifications wer e categorized into one of th re e groupings: "STD" (standard) fo rthose to be distributed to technical personnel such as subsystem manag ers and technicalmonit ors but not to pro gra m management, "Management" fo r those that requi red dis-tribution to MSC prog ram management and subsy stem man age rs and technical mo nito rs,and "Flash" for those recommended for forwarding by MSC program management toNASA Headquarters."Management. I ' Each notification was marked "N" fo r "noncritical" o r " P r r for?'program. '(

Prob-

The sa me distribution was made fo r "Flash" notifications as for

R o u t i n e N o t i f i c a t i o n and P r o b l e m S t a t u sA cent ral point at MSC managed all problem data. Distribution of data, includingcopies of problem re po rt s and resolution information, wa s made by thi s cent ral point.The hardware supplier sent to MSC all reportable p roble ms, including those fo r whicha 24-hour report had been submitted. Real- time displa ys of open prob lem data affectingthe next scheduled manned spacecraft launch were maintained by the central point for

use by the program manager . The hard ware su pplie r periodically submitted to MSC alisting of all open prob lems. Per iod ic listin gs of open pro ble ms wer e published anddistributed by the cent ral point. The se list ings provided probl em status informationand indicated the applicability of the prob lems to par tic ula r s pac ecr aft . Commencing60 days before each manned sp ace cra ft launch, a chart was prepared by the centralpoint showing the number of probl ems that wer e consid ered applicable to the vehiclesor supporting equipment f o r that mission. The cha rt was prepar ed weekly and updateddaily after the Headquarters FRR and distributed to approp riate MSC program manage-ment . An example of the ch ar t is shown in figur e 6.6


11/23

D A T A ELEMENT MATRIXTh e fol lowing matr ix indicates elements of data that the hardware suppl ier shal l , as a ininlmum,report toMS Cf or each nonconformance.The hardware suppl ier s! laI l record, and retainfor a t ime prescribed by contract , al l relevantdata for each nonconformance.

/ / / / /NO YES YES YES YESniquely Ident i f iab le Report Number

Date of nonconformance occurrence, or datenonconformance was detected i f occurrence YES YE S YES YES YESdate i s i i idetermlnable

. -

Ind icat ion of whether nonconformance isclassif ie d as fai lure or i insat isfactory YES YES YES YES YE Scondit ionPart number on which nonconformance occurred YES YE S YES YES YES

YES YES YES YES YESart name on which nonconformance occurredSerial number of part on which nonconformance YES YES YES YES YESoccurredManufacturer of part on which nonconformance NO YES YES Y ES Y ESoccurredSymptom of nonconformance YES YE S YES YES YESTes t being performed at t ime of occurrence YES YES YES YES YESBrief , narrat ive descr ipt ion o f nonconformance YES YE S YES YES YESEnd i tem on which nonconformance occurred, YES YES YES YES YE S

-~-

- ~ _ _ _ _~~~

~~_ _ _ _ ~ _ _ _ _ _ _ .

___.___

YES YES YES YES YESrevalent condit ions at t i me of occurrenceAl l end i tems whlch may be af fected by NO YES YE S YES YESnonconformanceProblem report numbers, and dates, that NO NO YES YES AArelate to the same problem

_ _ _ _ ~ ~ _ _~ -______Cr i t ica l i t y w i th re la t ionsh ip to m iss ion YES YES YES YES YESeffects (s ee Attachment C)Indicat ion of whether nonconformance is design IK IK YES Y E S AAoriented or manufacturing orientedAnalysi s results, includi ng laboratory test NO IK YES t AAresu l t sCause of nonconformance IK IK YES t A ACorrect ive act ion NO IK YES N/A AAPlanned date of disposit i oning NO YES NO NO YESExplanat ion rat iona le N/A N/A NO YES NOAssurance that explanations using redundancyan d/a alterna te modes of operation as one of the NO NO NO YE S NOelements do not neqate each otherWhen last test of art ic le, pr ior to mission, is tobe perf ame d. Statement as to whether or not non- NO NO NO YES NO

conformance is detecta ble during missionEf fect on missio n i f nonconformance recurred and N O NO NO YES NOrecommended owrational workaround orocedures

-

Previous history of nonconforminq art ic le NO NO YES YE S NOt

A AN/A Not Appl ica bleIK I f Known

Hardware suppl ier sha l l indicate any f indingsAs Avai lable, pr ior to resolut ion

Figure 3. - Example of data element matrix f o r m .

7


12/23

NASA - MANNED SPACECRAFT CENTERFAILURE INVESTIGATION ACTION REPORT NO.

1. PROJECT 2 . WHERE DETECTE D 3. ORG. REPORT NO. 4 . PROB. CLASSIF.F A C I L I T Y O R G A N I Z A T I O N L O C A T I O N 0 F A I L U R ER UNSAT. COND.

5. D A T E R E P OR TEI

6. CONTRACTOR 7 . END ITEM NAME 8. ITEM UNDER TEST 9 . NE XT ASSY. NAME 10. REP ORTED ITEM11 . TPS NUMBER 7 a. El MODEL NO. Ba. C ON TR . P A R T N O. 9a. C ON TR . P A R T N O.

22 CRITICALITY23 IN ITIATOR CONTACT ORG DA TE 24 R I E ORG D A T E

- 25 HARDWARE ANALYSIS REQUESTED INSTRUCTIONS

26 ASSIGNED TO ORG DA TE 27 REQUESTER ORG DA TE~ - _ _28 CAUSE OF FAILURE ANALYSIS RESULTS

loa . C ON TR P A R T N O.

29 . SYSTEM ENGINEER ORG. DA TE 30. RIE ORG. DATE

31 . CORRECTIVE ACTION REQUESTED

12 . ROUTING VIA 7b . El SERIAL NO. Bb . SUPPLIER PA RT NO. 9b . S U P P LI E R P A R T N O.

32. AC TIO N ASSIGNED TO ORG. DA TE 33. REQUESTER ORG. DA TE34. CORRECTIVE ACTION TAKEN

lob. SUPPLIER PART NO.

~ ~~~35 ACTION BY ORG DAT E 36 RIF ORG DATE p? C L OS E - OU T D A TE

13. SPEC PROCESS NO BC . SERIAL NO. 9c . SERIAL NO.D A T E P A R A.

I IASC FORM 2174 (JUL 6 6 ) P A GE- F-

Figure 4. - Example of fa ilure investigation action report .

1Oc. SERI AL NO.

8


13/23

0 F A I L U R E 0 UNSAT COND. 0 UNDETERMINEDPROBLEM NOTIFICATION

SSM FLASH- TD.- SC LOG NO.SSE MAN AGE MEN T REPORT NO.0 CSM 0 L M 0 G&C PAEASSESSMENT N P PAC CODE (NR)ORIGINATOR LOCA TION PHONENONCONFORMANCE SITENONCONFORMANCEOCCURREDDURINGDATEOFNONCONFORMANCE SUBSYSTEM.D A T E/T IME R EC E IVEDCONTRACTOR SUBCONTRACTORPA RT NUMBER/SN

S/C OCCURRED ON OR FIRST AFFE CT ED

CRITICALITY

P A R T N A MENE XT HIGHER ASSEMBLY TP NO.NONCONFORMANCE DESCR I P T lON

~~ ~

SUSPECTED CAUSEDISTRIBUTION STANDARD, PLUS

RECEIVED BYPRO BLEM ASSESSMENT ENGINE ERING

Figure 5. - Example of problem notification report.9


14/23

N - N o -l0l

9l

ll

0lh

w%0

10


15/23

P r o b l e m C l o se o u tThe MSC was required to concur in al l contractor problem closeouts. The prob-

le m cont rol sh eet shown in figu re 7 was used to rec or d MSC acceptance of closeouts .Signatures were required by the cognizant MSC design engineer and the SR&QA engineer.The problem wa s closed o r explained.lem control sheets. The OP L was updated to reflect the actions taken, such as completeremo val of a problem if it was resolved for the entire p rogram o r indication of thespac ecra ft to which the resolution was applicable.

Contractor s wer e provided copies of the prob-

P o t e n t i a l H a r d w a r e I m p a c t P r o b l e m sThose problems that were considered to have a potential hardware im pact wer emarked on the OPL with the letters "PHI. '' If known, the date the nonconformance wasdeemed to have a potential hardware imp act w a s al so noted. When applicable, the end-it em s affected wer e indicated.

M a n a g e m e n t R e v ie w sA series of selected contrac tor and customer management reviews w a s conductedat various l eve ls of management throughout the pro gra m to discipline the sys tem and

accelerate problem closu res.The Apollo hardware problem experience from program initiation to mid-1972 is

shown in figu re 8. Mor e than 50 000 problems wer e experienced during the cours e ofthe Apollo Pr og ra m. The slope of the cumulative problem cur ve is very sensitive topro gra m activity. The peak in prob lems occ urre d in ea rl y 1967 (peak of certif icationtest activity), and a gradual slowdown in activity oc cur red in e ar ly 1969 (completion ofmos t of the certif icat ion test activity and subsystem deliver ies).

An evaluation was made of vari ous problem caus es. The re su lt s are plotted infigure 8(b) for the Apollo spa cecraft and in figure 8(c) fo r the Apollo sp acec raft groundsupp ort equipment.F r o m figure 8(b), it can be calculated that m ore than 18 pe rc en t of the Apollo

spacecraft proble ms were from design causes; more than 35 percent were due tomanufacturing/procedure causes; and approximately 20 perc ent wer e due to human er-ror. Simi lar calculations fo r ground support equipment can be obtained from figu re 8(c).

Another significant item evident in figure 8(b) is that, although the majority of thespac ecra ft design and human er r o r problems occurre d in 1966, the manufacturing/the certification test prog ram was completed and a trained checkout team was fully op-erat iona l by 1967. However, even though manufacturing was significantly redu ced duringthis period, a la rg e amount of rework (because of des ign changes) added substan tial ly tothe manufac turing and test ing level of effort. This may explain the higher incidence ofmanufacturi ng/procedre problems. The problems experienced fro m 1963 to 1972 forvar iou s Apollo hardwar e (command and servi ce module (CSM), lun ar module (LM),guidance and navigation (G&N), and Government furnished equipment (GFE)) are shownin fig ur es 9(a) to 9(g). The figu res again emphasize peak problem activity in 1966.

proc edur e prob lems continued. This is explained, in part , beca use the maj or part of

11


16/23

PROBLEM CONTROL SHEET

R&Ir0A

C H EC K A L L A P P L I C A B L E B L O CK S R E P O RT NO. (7)

Rel i ab l i l i t y (14) (12)(1 ) (12)ual i ty ~--

(8)1)0 LO SEO U T ( 6 ) a E E E P A R T S,A .

SU B SYS/F lLE N O.(2) [7 E X P L A N A T I O N

1

( Y 1P A R T N O .3)0 E C H N I C A L A G R E E M E N T(4)0 E S I G N - OR I E N T E D P R O B L E M(5)0 A N U F A C T U R I N G - O R I E N T E D P R O B L E M A P P L I C A B L E E N D I T E M (S ) (10)

I S A F E T Y I (16) I (12)Figure 7.- Example of problem control sheet.

1 2


17/23

60 lo3

-ib5 12nc

I

1 6 /---Block I missions ,/-Design /-16--

/-- - Human er ror- -- Manufac tur ing lprocedure/

Spacecrafl hardwareICSM. LM, G&N, GFE)

1

20 0s0 160aLm-z 205e= 80 -eS 4 0 -Ea

I

50e

---

Calendar date

(a) Total.

Calendar date

(b) Spacecraft.

* I o3- DesignHuman er ror- 31 ------ Manufac tur ing iprocedure ,----

an

I I I I I I I I I I I17e Block U missions1963 1% 5 1%7 1969 1971 1973

r

m

Block II missions

I l l ]1971 1973

b 1200 -

1%3 1%5 1967 1%9

eL3k 01%3

O L1% 3

Calendar date

(a) Total.

-6Block 1 missions17Block II missions

1965 1%7 1969 1971 1973Calendar date

(b) Lunar module.- I I I I I I I I I I1 6 7 17Block I missions Block n missions

Calendar date Calendar date

(c) Ground support equipment. (c) The LM ground suppor t equipment.Figure 8. - Apollo hardware cumulative Figure 9. - Apollo hardware problemproblem experience. experience.

13


18/23

-620 -EL

17Block Il missions

1 1 1 II 1%3 1%5 1967 1969 1971 1973

20 050 160m.A-g 1 2 0L5= 8 0 -emL

4 0 -PI

Calendar date

-

-

-

(d) Command and serv ice module.

DL

L3

1963

I I I I I I 1 1 I 1 1

Block n miss ions 17

Block 1 missionsI I I I

1965 1%7 1969 1971 1973

5 320 u6Block I missionsCalendar date

(f) Guidance and navigation.u 1 I l I I I I I I J1 6 1 17Block I missions Block Il missions

01 1 1 / I I 1 1 I I 1 11%3 1965 1967 1969 1971 1973Calendar date Calendar date

(e) The CSM ground suppor t equipment. (g) Government furnish ed equipment.Figure 9 . - Concluded.

Subsystem breakdowns denoting where the pr oble ms wer e detected are shown intable I. These subsystems a r e listed in orde r by percentages of total spacecr aft hard-war e failures found in each subsystem. The majority of the failur es were in the elec-trical subsystems; the percentages found in screenin g tes ts were especially high.Because of difficulties in obtaining components tha t would meet mi ssio n req uire ment s,special attention was given to an electrica l, electro nic, and electromechanical (EEE)pa rts evaluation program. In the pri mary mechanical subsystem, the overall failurera te s and the percen tages of sc reenin g test fai lu res were relatively low, while the per -centages of problem s found in certification testing w ere generally gr eat er than the mean.

1 4


19/23

T A B L E I . - SUBSYSTEM FAILURE DETECTION[Data a s of Sep temb er 19721

S u b s y s t e m

I n s t r u m en t a t i o nE n v i r o n m en t a l co n t r o lD i s p l ay s an d co n t r o l sP r o p u l s i o nR eac t i o n co n t r o lE l e c t r i c a l p o w e rC o m m u n i ca t i o n sS t ab i l i za ti o n c o n t r o lFu e l ce l l an d c r y o g en i c sGuidance and con t ro lS t r u c t u r e s an d m e c h a n i c sE a r t h l an d i n g an d u p r i g h t in gC r e w e q u i pm e n tO r d n a n c e

M e a n p e r c e n t

I-Su b s y s t em f a i l u r e s b r eak d o w n ,p e r c e n t

T o t a l f a i l u r e s ,p e r c e n t15 .9

1 3 . 91 2 . 41 0 . 810 .4

7 .67 . 44 .84 .73. 92 .82 . 01 .81 . 6

E l e c t ri c a l a s s e m b l i e sP r o p u l s i o nI n s t r u m en t a t i o nE n v i r o n m en t a l co n t r o lRadarR eac t i o n co n t r o lC o m m u n i c a t i o n sA b o r t g u i d an ceS t ab i l i za ti o n co n t r o lE l e c t r i c a l p o w erS t r u c t u r e s a n d m e c h a n ic sC r e w e q u i pm e n t

M e a n p e r c e n t

18 .61 5 . 112 .5

9 . 17.67 . 66 .96 .25. 74.34.12 .2

CSM77. 372. 571.972 .460. 255. 768 .177 . 866 . 061. 536.246.934. 74 6 . 060.6

LM7 4 . 069 .959.06 2 . 676 .457.577.26 8 . 266. 537. 332.942. 560. 3

V eh i c l eC er t i f i ca t i o n Iest 1 Pr e f l i g h t3. 9

1 2 . 21 0 . 21 9 . 415 . 324. 51 8 . 91 5 . 122 .71 6 . 436. 031 .65 1 . 24 4 . 722.9

1 6 . 01 1 . 413 .814. 315 . 41 3 . 91 6 . 81 6 . 125. 326.824 .433.61 9 . 0

18 .413 .91 7 . 2

8 . 124. 01 8 . 811 .6

5.99 . 7

1 9 . 526. 51 9 . 61 2 . 1

8 . 715. 3

9 .51 8 . 626 . 722 .7

8 . 227 .5

5.915 . 3

7.735. 342. 523. 320. 3

0.41 . 4.7.1. 5

1 .01 . 41 . 21 . 62 .61 .31 .92. 0

. 61 . 2

0 .5.1. 5. 4. o

1.1.1. 4. 5. 6. 2. 6. 4

ab I n c l u d es p o s t f l i g h t f a i l u r e s f o r C SM o n ly .

A c c e p t an c e t e s t procedure/preinstallation t e s t .

1 5


20/23

PERFORMANCE OF THE PRO BLEM REPORTINGA N D C O R RE C TI VE A C T I O N S Y S T E M

As can be seen in figure 9(a), the rate of prob lems occurring in the program ro severy steeply in 1965 and 1966, peaking at 1800 p e r month in late 1966. To handle thisquantity, sever al means w ere use d by NASA and the cont racto rs to clo se the problemsexpeditiously. Handwritten duplicate logs of open pro ble ms we re maintained a t MSC andat the prime contractor plants to track the open problems. Daily telephone conferenceswe re held between cognizant NASA and contr acto r engineering personnel to di scu ss openproblems and develop means for closeouts. On criti cal problems, special team s ofNASA and cont racto r personnel we re est ablished to work the problem in "rea l time.This involved, in som e cases, hand-carrying the failed hardware to the vendor fo r fail-u r e analysis, witnessing the failu re analysis, and expediting the paperwork fo r problemcloseout. A s a resu lt of these eff orts and the corre ctive actions leading to hardwarematurity, the number of problems dropped dramatically in late 1966.

CONCLUD I N G R E M A R K SThe Apollo P rog ram was a ver y complex pro gra m undertake n by NASA. The si zeof the pro gra m dictated that a new approach was nec essar y to understand and cor re ct

prob lems . This was not initially recognized, and a s e r i e s of changes took plac e to cor -re ct the deficiencies in the P roblem Reporting and Correct ive Action System to developit into it s present form. The rate of oc curr ing prob lem s was ver y high fr om mid-1965to lat e 1966. Means use d to evaluate and clos e this lar ge number of pr obl ems includedhandwritten tracking logs, daily telephone conf ere nce s between cognizant NASA and con-tra cto r personnel, and formation of special task team s to work on critica l problems.The fact that such a lar ge number of probl ems were closed is an achievement worthyof note. Fea tur es of the Prob lem Reporting and Cor rec tiv e Action System use d in theApollo Progra m a r e applicable to future manned spacecraft, but ca re should be exer-cised to adapt the sys tem to the different requirements .

RECOMMENDATI ON SOne recommendation fo r the design of t he Pro ble m Reporting and Cor rec tiv e

Action System for future pr ogr ams is to change the philosophy of requ irin g that ev eryopen problem be res olved before flight. For prio rity of work purposes, it may be pos-sibl e to categ orize the problem s by the criti cali ty of the equipment involved and u s e theApollo explain technique. On less criti cal equipment, problem analysis re late d to actualteardown of har dwar e may be dictated by tre nds of o ccu rre nce r at he r than by ana lys isof each problem as it occurs.

Another possibility is the establ ishment of a problem an alysis facility at the launchsite. In the Apollo Program, most of the failed hardw are was retu rned to the vendor forfail ure analysis, with attendant delays in shipping and in ensurin g that a n adequate analy-sis was performed by the vendor. This may not be feasible on future programs .

16


21/23

The re a r e possible applications of the Apollo spa cecra ft Pr obl em Reporting andCor rec tiv e Action System to futur e manned pro gra ms other than those outlined. Manyof the Apollo guidelines can be applied to ensure that the hardware launched as a par tof the United States space program is adequate f o r mission performance.Lyndon B. Johnson Space Center

National Aeronautics and Space AdministrationHouston, Texas, November 5, 1973951-18-00- 0-72

1 7


22/23

A P P E N D I XDEFl N T 1ONS

Words o r terms used in this report that have a special connotation o r meaning arelis ted below.Nonconformance: Nonconformance is a condition of an y article, material, o rse rv ic e in which one o r more character istics do not conform to requirements. This

includes failures, unsatisfactory conditions, d iscrepan cies, deficiencies, defects, andmalfunctions.Problem: A problem is any nonconformance that fits, or is suspec ted of fitting ,into one of the following categorie s: (1)failure o r unsatisfactory condition occurr ingduring o r after production acceptance testing or (2) fa i lure o r unsatisfactory conditionoccurrin g before acceptance testing that will or has the potential to affect safety ad-ver sely , contribute to schedule impact, cause a launch delay, or res ult in design change.Failure: Failure is the inability of a system, subsystem, component, o r par t t operform its required function within specified lim it s under specified conditions for aspecified duration.Unsatis factory condition: An unsatis fac tory condition is a n y defect for which en-

gineering disposition is required and which requires recurrence control beyond thespecific arti cle under consideration. Included in thi s definition a r e conditions that can-not be corrected to t h e specified configuration using the standard planned operations o revents that could lead t o a fail ed condition but do not affect the function of the ar ti cl e,such as contamination, cor rosi on, workmanship requ iring engineering disposition, andso forth.

Open problem: An open problem is a problem f o r which responsible MSC manage-ment personnel have not approved the problem resolution submitted by the hardwaresupplier. A problem is deemed to be open until the ha rdw are supplier is formallynotified by MSC that resolutions a r e acceptable fo r all delivera ble end-item s t o whichthe problem is applicable.

Resolved problem: A resolved problem is a problem that has been closed orexplained.Routine problem: A routine problem is a problem that has no potential to affectthe program adversely.Closed problem : A problem is closed when the hardware supplier is formallynotified of MSC concurrence with the problem analy sis (including dete rmination of the

cause) and with the implementation of cor rect ive action to preclud e re cu rr en ce of theproblem on hardware after acceptance tests. A lack of corrective action may be ac-ceptable to MSC f analytical and test evidence from the hardw are supplier shows thatthe problem is always detectable during the perfo rman ce of an established test beforeend use, and that the problem will not occur afte r thi s test .

18


23/23

Explained problem: A problem is explained when the hardware supplier is for-mally notified of MSC concur ren ce with the problem ana lys is and rationale for not es -tablishing cor rec tiv e action. The rationale must esta blish t h a t a planned mission mayproce ed with no detrime ntal eff ect s should the problem re cu r and t h a t a MSC contrac-tually responsible authority (that is, the Configuration Control Board) has decided thatno correct ive action, as defined for a closed problem, need be established.

Problem analysis: A problem analysis is documented re su lt s of the investigationperformed to determine the cause of the problem.Cause (problem cause): The cause o r problem cause is the event or se ri es of

events directly responsible for the problem.Corrective action: Corre ctive action is action established to prevent recurrenceof the problem.

apollo experience report problem reporting and corrective action system

Documents