eliciting engineering knowledge about reliability during design-lessons learnt from implementation

QUALITY AND RELIABILITY ENGINEERING INTERNATIONAL

Qual. Reliab. Engng. Int. 2001; 17: 169–179 (DOI: 10.1002/qre.409)

ELICITING ENGINEERING KNOWLEDGE ABOUT RELIABILITYDURING DESIGN-LESSONS LEARNT FROM IMPLEMENTATION

R. HODGE1, M. EVANS2, J. MARSHALL2, J. QUIGLEY1 AND L. WALLS1∗1University of Strathclyde, Glasgow G1 1QE, UK

2TRW Aeronautical Systems, Lucas Aerospace, Birmingham B28 8LN, UK

SUMMARYIn electronic design the use of engineering knowledge and experience is considered important in understandingand estimating the reliability performance of complex systems. There are numerous methods proposed for elicitingthis knowledge in order to ensure that the data collected are valid and reliable. In this paper we describe ourexperiences in implementing an elicitation process that aims to extract engineering knowledge about the impact ofdesign changes on a new aerospace product that is a variant of an existing product. The elicitation procedures usedwill be outlined and the ways in which we evaluated their usefulness will be described. This research generatedmany useful insights from the engineers and facilitators involved in the elicitation exercise. This paper shares theirperspectives on the gains and losses associated with the exercise and makes recommendations for enhancing futureprocedures based on the lessons learnt. Copyright 2001 John Wiley & Sons, Ltd.

KEY WORDS: elicitation; expert knowledge; lessons learnt; design for reliability

1. INTRODUCTION

It is generally accepted that designers should bemore involved in the reliability assessment of theirsystems as they have considerable experience tocontribute [1,2]. We subscribe to this view and intendto show how this might be achieved within the contextof modelling the reliability of electronic systemsduring a variant design process. Our ultimate aimis to combine soft engineering knowledge about theperceived reliability of a new design with hard failuredata relating to actual performance of a similar systemusing the REMM† model so that the reliability of thenew design can be estimated and better understood.Details about the REMM modelling process are givenin [3], for example. In this paper we focus on theelicitation of engineering knowledge and share ourexperiences of implementation within a large projectundertaken at TRW.

Many papers have been written about proceduresfor eliciting expert knowledge within risk assessment

∗Correspondence to: L. Walls, Department of Management Science,Strathclyde Business School, Strathclyde University, 40 GeorgeStreet, Glasgow, G1 1QE, UK.†REMM is a collaborative research project involving aerospacecompanies (TRW, Smiths Industries, Marconi Electronics, BAESYSTEMS), the RAF and the Universities of Strathclyde, Warwickand Loughborough. REMM is part funded by the UK DTI underits More Electric Aircraft Challenge and part funded by theaforementioned aerospace companies.

(e.g. [4–6]) and the books by Cooke [7], and Meyerand Booker [8] provide comprehensive guides togeneral aspects of elicitation. However, very littlehas been reported about elicitation processes aimedspecifically at gathering data from a design teamand even less has been written about the experiencegained from the different people involved in theprocess.

The elicitation process described in this paperaims to assess the impact of design changesbetween variants of a product both qualitativelyand quantitatively. The process has been stronglyinfluenced by the elicitation theory but is designed tobe robust and useful in the environment in which it isapplied.

The REMM elicitation process was carried out atTRW, Aeronautical System’s York Road site, duringlate 1999/early 2000. This site both designs andmanufactures high integrity, safety critical electronicsfor the aerospace industry. A current project met withthe requirements of the REMM elicitation exercise.This project is in the design stage and is derived froman existing product, which is in service.

The elicitation exercise involved teams from bothREMM and TRW. The aims of these project teamsoverlapped; for example, the REMM team aimed toevaluate the elicitation process and the TRW projectteam aimed to gain a list of project concerns.

Received 28 November 2000Copyright 2001 John Wiley & Sons, Ltd. Revised 28 February 2001

170 R. HODGE ET AL.

Figure 1. Roles within the elicitation exercise

Figure 1 illustrates the key roles within bothteams and within the elicitation exercise. Theactual implementation of the elicitation process wasconducted jointly by a member from each team.

Essentially the REMM modellers took responsi-bility for designing and developing the elicitationprocess and the evaluation procedures. The REMMproject manager co-ordinated activities between thetwo projects. The facilitators customized and imple-mented the elicitation process and members of thedesign team provided the expertise.

A cross section of the TRW design team wereinterviewed. It is important to include the correctpeople, in terms of both speciality and knowledge.The choice made reflected how the project wasorganized and the project stage. Representatives werechosen from electronic design, mechanical design,environment, process, manufacturing and componentengineering. The persons chosen were experienced(e.g. principal and senior engineers) as they wereconsidered to have a broader perspective of the projectand the necessary knowledge and aptitude to makeinformed and impartial decisions.

In the following sections of this paper themethodology underpinning the elicitation process andits evaluation is outlined. The challenges of ensuringthat valid and reliable data are captured that reflectthe knowledge of the design team will be discussed.The issues that arose during implementation areexplored by summarizing the experiences of thedifferent groups of people involved. Finally we reflecton the lessons learnt by all involved and makerecommendations for improvements to the elicitationprocedures.

2. PROCESS FOR ELICITING KNOWLEDGEFROM THE DESIGN TEAM

The objective is to capture the views of the designteam in terms of the concerns they have about thenew system both qualitatively (i.e. an engineering

description of a potential fault) and quantitatively(i.e. probability of that potential fault being realizedas a failure if no action is taken).

These concerns tend to stem from the changes madebetween design variants in areas such as componenttechnology, environment and, of course, the designprocess itself. Thus we need to ensure that all relevantdomains of engineering expertise and experience areincluded in the design team from whom data is to beelicited.

In adopting this approach we are assuming thatthe information about the reliability of the commonaspects of two variant designs is captured in the hardfailure data for the similar system and so is outside theremit of this paper.

Figure 2 illustrates the elicitation process used. Thebasic input is the proposed elicitation procedures. Theoutputs are the revised process in the light of thetest as well as preliminary data in the form of aprior probability distribution for input to the REMMmodel and the list of engineering concerns for TRW.The elicitation itself is divided into two phases. Thefirst is interviews with each individual engineeringexpert, after which there was a feedback and reviewsession that was followed by the second phase wheregroup sessions were held in order to explore potentialdependencies between experts’ views. The reason forphasing the elicitation was to improve managementof the process and to ensure better quality data arecollected.

2.1. Phase 1: Interviews with individual engineeringexperts

The individual elicitation sessions were designed inaccordance with the process described in Walls andQuigley [9], who developed their approach in line withthe theory of the SRI model [10] and reported theusefulness of the approach within a small reliabilitygrowth scenario. These preliminary findings indicatedthat the process was successfully designed to managethe many biases that can easily corrupt judgementaldata.

In summary, the individual sessions involvedcollecting preliminary data from the engineeringexpert and his initial concerns with the design througha self-completion questionnaire. This is followed upwith a semi-structured interview where the facilitatortakes the engineer though the changes made duringdesign relative to his domain of expertise. The failurecategories that may result from those changes and theareas of concerns he may have are identified. Theseconcerns are ordered with respect to their likelihood

Copyright 2001 John Wiley & Sons, Ltd. Qual. Reliab. Engng. Int. 2001; 17: 169–179

RELIABILITY ASSESSMENT AT THE DESIGN STAGE 171

Figure 2. Flowchart of the elicitation process

of occurrence and a probability is attributed to theconcern before a process of verification is carried out.The same information is then elicited by an alternativeroute so that the consistency of beliefs can beestablished. The engineer is asked to state how likelydifferent scenarios regarding the number of concernswithin each class of failure are to be realized. Theverification process involves providing the engineerwith visual feedback in terms of the two probabilitydistributions generated using a spreadsheet applicationespecially designed to support the elicitation process.Inconsistent beliefs are explored and, when agreementis reached, a prior distribution for the number ofconcerns for that individual engineer is recorded.

Further details about this phase of the process canbe found in Walls and Quigley [9].

2.2. Phase 2: Group workshops with potentiallycorrelated engineers

Since it is expected that different engineerswill share similar concerns, albeit from different

perspectives, it is important to conduct sessions withgroups of relevant engineers to explore this furtherand refine the data accordingly. This will ensure thatthere is no double counting of identical concerns andthat any correlated views are apportioned across thefailure categories appropriately. The process designedto manage the group sessions was informed by thetheory developed in Quigley et al. [11].

The group sessions have been designed to satisfythe following requirements:

1. to provide a mechanism to explore the reasoningunderpinning the engineers’ views of potentiallyrelated concerns so that they can be categorizedas identical, correlated or independent;

2. to capture the probabilities allocated to concernsonce they have been categorized following anopen and full discussion.

The chosen method that meets the dual aims offacilitating group management as well as providing amechanism for recording data is to construct maps ofconcerns as follows. Essentially for each grouping of


172 R. HODGE ET AL.

potentially correlated concerns a map is prepared withthese concerns centred on the page. The facilitator asksthe engineers to explain what factors lead to theseconcerns and these are mapped to all the relevantconcerns. Next the engineers are asked to explain whatcorrective actions might be taken to mitigate these sub-concerns. These in turn are mapped.

Figure 3 shows an example of such a map that wascompiled for demonstration purposes and is used atthe beginning of new group sessions to inform theengineers present of the process to be followed. Theconcerns ‘excess handling’ and ‘rework’ are the initi-ating events for this session, having been raised duringthe interviews by two different engineers and deemedto be potentially related. These two engineers arenow sharing their views in a joint session. Each liststhe factors that he believes impact on his concerns,and these are colour coded to distinguish the viewsof individuals. For example, engineer A (in blacksolid line) has identified four sub-concerns: ‘knockon failures’, ‘time taken’, ‘contamination’ and ‘cost’.The latter sub-concern is shared with engineer B,whose links are shown in grey. Having completed thisstage the engineers state what corrective actions theypropose are taken against the sub-concerns they haveflagged. For example, ‘operator training’ is regardedas a means of managing both ‘knock on failures’and ‘time taken’ by engineers A and B, respectively,while both engineers A and B recommend ‘designfor manufacture activities’ for managing ‘knock-onfailures’. In contrast, only engineer B believes that a‘change layout’ will correct the ‘scrap material’ sub-concern. These examples illustrate cases of correlated,identical and independent concerns, respectively, andprovide instances of the general rules that can beconstructed for categorizing concerns.

It is relatively easy for the facilitator to categorizeconcerns in real time. The engineers are then invited toallocate likelihood to each sub-concern being realizedas a failure for each category. In order to manageany reluctance from engineers to allocate singleprobability figures, we give them the opportunity toassign a likelihood range to each sub-concern. This ispresented to them in the form of a 0–100 horizontalscale with qualitative statements marked at variouspoints. For example, 0 implies ‘will never occur’and 100 implies ‘will certainly occur’. The engineerssimply have to mask on a point or range representingtheir belief, thus capturing their level of uncertainty. Adifferent likelihood scale is used for each concern toavoid the engineers anchoring to an original value.

To summarize, the group process, like the individualinterviews, was designed within the framework of the

SRI model [10] and so different stages are alignedto motivating, conditioning and coding the views ofthe engineers. The group sessions follow the basicprocess.

1. Groups of potentially correlated experts areidentified by the facilitator and sessions areorganized with them, preparing in advance mapswith relevant groups of concerns.

2. Within each session a mapping exercise asdescribed above is embedded within a processthat begins with the facilitator explaining thepurpose of the session and the process to be usedand that ends with a brief summary of the resultsobtained to quickly verify that the data capturedis an adequate representation of the group beliefs.

There are a few practical points to note in termsof implementing the above two stages. First, in termsof selecting groups, basically all concerns mentionedin the individual interviews must be considered. Theinitial consideration is to identify those concerns thatmay have been repeated, are in conflict and thosethat are deemed to be similar across individuals.These concerns are separated from the full lists sothat only those that are of primary interest to thegroup are explored further. Sessions are arranged toensure that the minimum number of groups are setup to consider the maximum number of concernsrelevant to the individuals in that group. We haveadopted a sequential group process to facilitate groupmanagement and to minimize the time demandsplaced on a design team. However, we are aware thatwe run the risk of ignoring concerns that might in factbe correlated although they might not appear so at firstsight. We believe that this risk is acceptable as it isunlikely that any manor concerns will be ignored.

Secondly, depending on the group size thefacilitator might adopt different tactics for themapping process. Two options for creating mapsthat both involve the engineers taking ownership fortheir beliefs are either by recording them on thesame master worksheet or recording them individuallyand attaching them to a larger worksheet. If thegroup is larger than about four then it likely thatas they identify individual sub-concerns their lackof involvement might cause them to lose interest inthe discussions. As a consequence all individuals canwork simultaneously before entering into discussionas to whether or not the sub-concerns were identical orsimilar. In smaller groups, say two or three engineers,each should work in turn recording their sub-concernswhilst identifying similarities in real time.



Figure 3. Example group map for a hypothetical manufacturing concern

2.3. Construction of prior distribution from dataelicited

The data elicited through the above two phasesrequires to be reconciled in order to provide a singleprior distribution of the number of concerns for theREMM model and a comprehensive list of engineeringconcerns that will contribute to both REMM and TRWrisk analysis.

The underlying stochastic process of the REMMstatistical model is based on a Bayesian orderstatistic methodology [12] and requires that theconcerns are partitioned into categories, such asdesign and manufacture. Not only does this allowthe impact of different factors on reliability to beestimated, but it also facilitates aggregation of theprobability data collected from individual engineersand through the group sessions. For example, usingthe data elicited we are able to construct priors fordesign concerns, manufacturing concerns and jointdesign and manufacturing concerns. The benefit ofpartitioning is that we can treat these priors asindependent and hence they are convoluted to obtaina distribution on the aggregate number of concerns. Afurther output of this partitioning process is a revisedlist of relevant engineering concerns.

3. EVALUATION STRATEGY

As a primary purpose of the elicitation exercise wasto test its usefulness with a view to identifying waysin which the process could be improved, an evaluation

strategy was embedded into the exercise at the outset.We aimed to ensure that full and relevant data on theperceptions of all people involved were captured in aneffective and efficient manner. To achieve this we useda combination of methods as summarized in Figure 4.

Short self-completion questionnaires were used tocapture the reactions of the engineers in the designteam at the end of each interview and group session.They aimed to collect feedback on issues groupedunder headings such as the general usefulness ofcapturing knowledge from the design team and thedegree of comfort with the interview or the workshop.Most questions involved marking a 5-point scale thatbest represented the respondent’s level of agreementwith a statement about the process. A mix of positive,negative and neutral questions were posed so thatthe validity and reliability of responses could bemeasured. The individual questionnaire also includedsome open-ended questions in order to prompt ideasfor improving the process that might have been missedduring the closed-question sections.

The choice of self-completion questionnaires forthe engineers was a deliberate attempt to minimizeadditional demands on their time. In contrast webelieved it more appropriate to capture feedback fromthe facilitators in a less structured way so that we couldbetter gain an insight into their experience. Thereforede-briefing questionnaires were prepared includingstandard questions about issues relevant directly tofacilitation, such as comfort levels, adequacy ofsupport materials (e.g. spreadsheet, mapping props)


174 R. HODGE ET AL.

Figure 4. Evaluation strategy

and time management, as well as perceptions about,for example, group dynamics. The modellers whowere not directly involved in these interviews orgroup sessions were, however, in a position to observethe process through interactions with the facilitatorsand so were encouraged to maintain a learning diaryof their reflections on how their plans were beingexecuted.

Together these data collection activities provide thebasis for capturing the views of the different peopleinvolved in the process that will now be shared andevaluated.

4. PERSPECTIVES ON THE ELICITATIONEXERCISE

Figure 5 shows a map that has been built up fromall the issues generated through the evaluation ofthe elicitation process. In the following sections wediscuss the major issues captured in the map. Webegin by reflecting on the elicitation process. Inthe following sections we discuss the major issuescaptured in the map. We begin by reflecting onthe elicitation process as viewed by the facilitatorswhose responsibility it was to implement. This isfollowed by a summary of the findings reported bythe engineers who took part in the exercise. Finally,

the modellers who developed the theory of the processreflect upon the implications of implementation forfuture procedures and research.

4.1. Reflections of TRW facilitator

As a facilitator it is important to understand boththe product and the concerns that people were raising,together with the process that was being undertakenand what the REMM team really desired. Acting asan intepreter, translating between the aerospace jargonof our world to that which could be recorded andunderstood by the REMM team was an importantfunction.

It was necessary to highlight the benefits that thedesign project team would gain from this research.People tend to be more responsive if they understandthat the task is of some use in the ‘real’ world ratherthan a theoretical exercise which they would neverhear of again!

Timing was an issue that was of concern from thebeginning. Participants, although happy to partake,were busy, and having them sit waiting for 15 min tolisten to an enthusiastic engineer was not acceptable!There is definitely a balance between encouragingsomeone to express their views and yet keeping themfocused on your objectives. This skill quickly matured.



Figu

re5.

Map

ofis

sues

gene

rate

dby

engi

neer

s,fa

cilit

ator

san

dm

odel

lers


176 R. HODGE ET AL.

Although this exercise was mainly performed by theREMM team, in future it will be entirely up to internalpersonnel to run the task. The choice of interviewer iscritical. As the information learned during the processis vital to the end result it is important that everyoneinvolved within the process is at ease. Worthwhile datawill not be acquired if either the interviewer is not ableto focus and lead the process, nor if the interviewee isuneasy about the process.

Grouping the concerns elicited during individualinterviews in preparation for group sessions requiressound knowledge about both the product and thetechnology behind it.

Facilitating a group session is completely differentfrom individual interviews. The interactions betweenthe group become more dominant than the interactionsbetween interviewer and interviewee. Therefore, it isnecessary to allow everyone in the group to havetheir say, but at the same time to let the groupreach a consented answer. It is also advised to limitthe number of participants to a maximum of four.Many more than this and the process risks becomingunmanageable and no longer worthwhile.

4.2. Reflections of REMM facilitator

More time should have been dedicated to briefingengineers and issuing the pre-interview questionnaireswhen planning the exercise. Better data would havebeen collected at this stage had some of the questionsbeen re-worded to ensure relevance to specifyengineering domains. Despite our good intentions wedid not fully implement our plans in the flurry of goinglive.

During the individual interviews engineers werenot always comfortable and willing to discuss thelikelihood of their concerns resulting in failure. Therewas a need to clarify that the likelihood values werenot the chance of failure in the field but a failure atany point in the project from this point forward ifno further engineering resource was assigned to thearea in question. To support this statement the term‘failure’ needed to be defined as any event that requirescorrective action to be taken, be it a failure in testor a redesign, in order for the system to functionto requirements. In order to generate the numericallikelihood value we adopted the approach of eliciting aqualitative statement and inferring a numerical value.This approach proved successful in that it got a valueagreed that could be used to initiate discussion.

It was also clear that the engineers were having diffi-culty distinguishing between likelihood of occurrenceand risk. This should not have been surprising given

that the data collected was to contribute to assessingproject risks. However, from a reliability modellingperspective, it was important that the likelihood wasobtained. Therefore it was imperative to continuallyensure they understood the terminology and were ableto distinguish between the two.

The likelihood scales used in the group sessionswere much better received than the approach usedduring the individual sessions even though theprinciples were the same. However, we neededto ensure that the engineers only expressed theirlikelihood after explaining the reasoning behind thevalue attributed so that successive values are notanchored empirically on the first value stated.

It is also necessary to ensure that the engineerfully accepts and agrees that his belief has beencaptured and is consistent. During the verificationstage of the individual interview the prior statisticaldistributions were fed back to the engineer and anexplanation of what they represent was given. Ifthe distributions were significantly different then thereason was identified, using terms and examplesfamiliar to the engineers. Usually the priors elicitedwere consistent and, on the few occasions wherethey were not, the reasons for the difference wereestablished and agreement reached.

Similarly, in the group sessions it was importantto establish consistency of likelihood values betweenengineers when identical concerns were beingconsidered. In most cases, the likelihood expressedby different engineers was similar. When significantdifferences did arise, then the reasons for differentviews were clarified. Usually this related to adifference in the knowledge set of the engineer.If concensus was not deemed appropriate then thedifferent likelihoods and reasons were simply noted.

During the group sessions we were required tomanage the different personalities of the engineerssimultaneously. This entailed trying to discreetlyrestrain discussions with some engineers to maintainfocus, whilst prompting others to engage in thediscussion. During the individual interviews this wasnot quite as difficult to manage.

The example shown in Figure 3 worked well inillustrating the process to groups of engineers. Theyvisibly relaxed as they understood what they wererequired to do during the session. The engineersseemed to identify with the visual mapping. As wellas getting them more involved, it seemed to providethem with a sense of ownership towards the views theyexpressed.

It was also important to maintain accurate recordsof the sessions. During the individual interviews



qualitative data was recorded in note form to maintainthe momentum of the interview. However, this runsthe risk of not capturing the data fully. Thereforeit is important to understand fully, and be seen bythe interviewee to fully understand, the concernsbeing discussed as well as the reasoning behind theseconcerns. The spreadsheet model used to facilitaterecording and visual feedback to engineers wasextremely useful in supporting the statistical aspectsof the interview, but requires enhancement in order torecord qualitative data better.

The facilitators had to be vigilant during the groupsessions to monitor the data being recorded. Thiswas easy insofar as the recording method involvedeach individual within the group using a differentcoloured pen. Therefore it became apparent that whenindividuals took it upon themselves to capture thestatements of others the process was being disturbed. Itwas our responsibility as facilitators to ensure that thisdid not happen, but if it did occur it was to be correctedin real time. In a similar vein it was important toensure that group members completed the likelihoodscales independently so that their own beliefs werecaptured.

There was a general feeling that the group sessionsran more smoothly. However, this is not too surprisinggiven that both the facilitators and the engineers hadbeen on a steep learning curve.

4.3. Feedback from engineers

According to their questionnaire responses, mostinterviewees believe that there is a need to elicittheir knowledge and use it to assess the risks andreliability of the system both during concept designand throughout the project lifecycle. There is generalagreement that sharing of views about the reliability ofthe design will be constructive. A proactive approachis necessary to unearth engineering concerns duringthe development of a new system, updating theoriginal concerns throughout the product lifecycle inorder to monitor progress.

There was a general view that intervieweeswould be more comfortable being interviewed inperson rather than interacting directly with elicitationsoftware. However, this is a hypothetical questiongiven that no self-contained software was employeddirectly with the engineers. Engineers found itacceptable that the REMM facilitator conducted allindividual interviews.

Typically, interviewees believed that their issues ofconcern and their likelihoods of occurrence had beencaptured by the interview process.

Interviewees expressed concern about the wordingof some of the questions asked, especially on thepre-interview questionnaire. Requests were madeto customize phrases for different engineeringspecialities.

Other important issues that arose relate to thescheduling of interviews. For example, in this exercisewe decided to interview each engineer individually,and the ordering of interviews was based on theavailability of the engineer for interview. However, ithas been suggested that it might be useful to scheduleinterviews according to functionality with, for exam-ple, process engineers being interviewed after design,component and manufacture. This would also assistin the management of potentially correlated concerns.

In general the engineers felt suitably prepared toanswer the questions in the group sessions and becamemore engaged as the session progressed. The jointfacilitation team and the working environment wereconsidered fit for purpose. The engineers believedthe questions asked were relevant and understandable.They believed that the facilitators were aware of theissues and the engineers’ uncertainty about them. Theengineers agreed that the common concerns helpedthem to think about issues not considered previouslyand they felt that the concern maps are a suitable wayof expressing their beliefs. The fact that there werenot too many concerns to be discussed in each sessionenabled discussions to be managed effectively. Theengineers stated that they did not censor their views inthe presence of fellow team members, although theywere happy to leave the identification of potentiallyrelated concerns to the facilitators.

4.4. Reflections of modellers

This exercise provided a platform to operationalizea process we had developed for eliciting priordistributions. We already had some experience inapplying part of the process on a much smaller scaleand so had identified ways in which the procedurescould be extended and the practical implementationimproved [9]. However, this exercise proved to bechallenging insofar as it involved more people, it wasto be applied to an important and high-profile projectand we aimed to capture data that would contribute totwo projects which had complementary requirements.

Managing the elicitation process from theory topractice has been a team effort. Between us wehave strived to ensure that the process meets all thetheoretical requirements for modelling and at the sametime can be practically implemented. Not surprisingly,concessions did have to be made in moving from


178 R. HODGE ET AL.

academic theory to industrial practice, but these havebeen managed so that the data collected have not beencompromised. For example, there are always likelyto be demands on project resources that we cannotcontrol and so we need to be responsive to adaptingplans around these. The need for good communicationbetween team members cannot be overemphasized.This is non-trivial to manage given the demandsplaced on people and the resources at their disposal.If possible we would always recommend face-to-face meetings between partners so that a commonunderstanding can be achieved more quickly.

We had invested considerable energy in thinkingthrough the design of the elicitation process, and sohanding it over to others to implement has been bothstressful and exciting. Like any situation where oneloses control there is an element of risk regardingthe outcome. The obvious way to minimizing thisis to ensure that all partners on the hand-over arefully prepared. If this is achieved, then delegating theimplementation to facilitators, who are better placed toimplement the process anyway, can be very satisfying.To see ideas being developed and made useful is, afterall, what we aspire to. However, lack of preparationon our part can cause problems downstream. Therewas an occasion where we were required to takesome small corrective actions to keep the process ontarget, the basic problem being a simple oversightin the guidelines on the facilitators. Certainly goodpreparation always paid off downstream.

As well as thinking through the process we alsorecommend that researchers become involved intesting the process by assuming the role of experts. Wefound this to be a very valuable exercise. It providedgood training for the facilitator and it also allowedus to be the beneficiaries (or victims?) of our owninstructions. We were led to make radical changes tothe way the group sessions were managed in the lightof this experience—for example, using simultaneousrather than sequential methods of capturing data fromthe group. This was in many ways a simple change,but it did make a significant difference to our degreeof engagement in the exercise. It is amazing how easyit is to suspend judgement about a process you knowwell (at least theoretically) and engage in a practicalexercise when it is being ably led by a facilitator.

We have not made, nor do we intend to make,any major changes to the fundamental theory of theelicitation process. All the evidence suggests that thisis not necessary. However, there are many ways inwhich the process can be enhanced based on thefeedback from all involved, much of which has beenoutlined in the above reflections.

5. LESSONS LEARNT AND FUTURE WORK

During this elicitation exercise we have learnt manylessons that we intend to address in future. We sharethe major lessons learnt by the elicitation team whichwe believe may also have more general currency.

• It has been established that a design team demon-strated commitment to sharing views and con-tributing to a reliability enhancement activity.This is evidenced by their responses to question-naires and, more importantly, their involvementin this exercise. Therefore in future we wouldseek to initiate more interactions between thedesign team and reliability engineering.

• Good preparation is fundamental to implement-ing an elicitation process in practice. The effortrequired to ensure adequate preparation shouldnot be underestimated. Any shortcuts during ini-tial preparations will most likely cause problemsdownstream. We found this to be true, althoughfortunately the consequences were not seriousand minor corrective actions kept the exercise ontarget. In future, we will never compromise ourpreparation. Instead we would negotiate reviseddeadlines if required so that sufficient time isavailable to prepare adequately.

• It is important to fully brief engineers takingpart in the elicitation about the purpose andgoals of the exercise. In future we would seekto ensure that the planned group presentation isimplemented well in advance of interviews asthis is considered by all to be the most effectivemeans of communication. In addition, we woulduse more relevant examples to increase the buy-inof participants.

• The process by which we select engineersto be interviewed will be revised. Althoughthis exercise did include a representative cross-section of engineers, it was biased towardshardware and so excluded software engineers aswell as those engineers who have experienceof similar products but who are not part of thecurrent design team.

• In future there is only likely to be one facilitator,and that is likely to be a reliability engineerassociated with the design project. This isappropriate as he is familiar with both the contentof the design and the process of the elicitationprocedures.

• The challenges of capturing valid probability dataneeds to be revised in line with our findings.We aspire to ensuring that the method used tocapture probabilities engages the engineers. It is



clear that this is an area with which they feelless comfortable and yet it is vitally importantfrom the perspective of REMM modelling. Giventhe relative success of the maps used duringthe group session, we are reviewing how ourprobability capture in the individual interviewscan be enhanced.

• The methods of recording data worked quitewell. The spreadsheet model used during theindividual interviews and the maps used duringthe group session provided useful tools. However,both were prototypes and a number of ways ofenhancing the interface to facilitate a smootherflow during the sessions are to be implemented.

• Throughout the exercise we have accumulatedconsiderable experience that should inform futureimplementations. This experience covers themajor issues discussed above as well as morefundamental points, such as using the appropriatelanguage when discussing reliability with adesigner. Both have very different perspectives!There is no perfect elicitation process. Simplyplanning what is best for a particular context andthen gaining experience through use is a suitableway forward.

In terms of future work, the REMM team areusing the results obtained from the reported elicitationexercise to develop better supporting elicitationsoftware that can be incorporated with the REMMdemonstrator [13]. The prior distribution for thenumber of engineering concerns constructed is beinginput to the REMM statistical model as part of itsinitial validation. The engineering concerns have beenused by the TRW project team as part of their riskassessment, and aspects of the elicitation processwill be included in figure company procedures. Moregenerally the exercise has demonstrated that designteams are willing and able to contribute to reliabilityactivities and the information they provide can be usedto support reliability enhancement.

ACKNOWLEDGEMENTS

We would like to acknowledge the input fromengineers who took part in this exercise, namely Jim

Egan, Mike Pupins, Ann Crompton, Andy Medly,Mike Slater, Rob Travis, Dave Gardner, Steve Upton,Ian Fox and Cliff James.

We would also like to acknowledge the financialsupport of the UK DTI and the financial andtechnical support of TRW Aeronautical Systems,Smiths Industries, Marconi Electronic Systems andBAE SYSTEMS.

REFERENCES

1. Denson WT, Keene S, Caroli J. A new system reliabilityassessment methodology. Proceedings of Annual Reliabilityand Maintainability Symposium, Anaheim, CA, 1998; 413–420.

2. Kerscher W, Booker J, Bennet T, Meyer M. Characterisingreliability in a product/process design assurance program.Proceedings of Reliability and Maintainability Symposium,Anaheim, CA, 1998; 105–112.

3. Walls LA, Quigley JL. Learning to enhance reliabilityof electronic systems through effective modeling and riskassessment. Proceedings of Reliability and MaintainabilitySymposium, Los Angeles, 2000; 358–363.

4. Bubniz RJ, Apostolakis G. Boore DM, Cluff LS, CoppersmithKJ, Cornell CA, Morris PA, Use of technical expert panels:Applications to probabilistic seismic hazard analysis. RiskAssessment 1998; 18:463–469.

5. Rosness. Risk influence analysis: A methodology foridentification and assessment of risk reduction strategies.Reliability-Engineering and System-Safety 1998; 60:153–164.

6. Villain B, Verite B, Biernacki C, Celeux C. A practicalapproach to expert elicitation for Bayesian reliability analysisof aging. Proceedings of the European Conference on Safetyand Reliability, Munich, Germany, Scheuller GI, Kafka P(eds.). 1998; 13–17.

7. Cooke R. Experts in Uncertainty. Oxford University Press:Oxford, 1991.

8. Meyer M, Booker J. Eliciting and Analysing ExpertJudgement: A Practical Guide. Academic Press, 1991.

9. Walls L, Quigley J. Eliciting prior distributions fromengineering experts to support Bayesian reliability growthmodelling. University of Strathclyde Working Paper, 1998.

10. Merkoffer M. Quantifying judgemental uncertainty: method-ology, experiences and insights. IEEE Transactions on Sys-tems, Man and Cybernetics 1987; 17:741–752.

11. Quigley JL, Walls LA, Hodge R. Eliciting prior distributionsfor potential system faults from correlated experts. Foresightand Precaution, Cottam MP, Harvey DW, Pape RP, Tait J(eds.). The Safety and Reliability Society: Edinburgh, 2000;325–329.

12. Quigley J, Walls LA. Measuring the effectiveness ofreliability growth testing. Quality and Reliability EngineeringInternational 1999; 15:87–93.

13. Marshall J, James I, Jones J. Reliability enhancementmethodology and modelling. Foresight and Precaution,Cottam MP, Harvey DW, Pape RP, Tait J (eds.). The Safetyand Reliability Society: Edinburgh, 2000; 1479–1487.


eliciting engineering knowledge about reliability during design-lessons learnt from implementation

Documents