expert-opinion elicitation

“ Building Strong “

Delivering Integrated, Sustainable, Water Resources Solutions

Expert-Opinion Elicitation

Robert C. PatevNorth Atlantic Division – Regional Technical Specialist

(978) 318-8394



• Subjective Estimation• Elicitation Process

– Background– Expert-Opinion Elicitation (EOE) Process

• Probability– Axioms of Probability– Medians and Percentiles

• Training Example




Subjective Estimation

• Uses of one or more experts to estimate a probability (qualitative or quantitative) for use in engineering risk analysis– Good for first estimate of probabilities– Quick, cost effective and efficient method– Problems:

• Not a formal elicitation• Usually not well documented• Probabilities may not be repeatable or defendable• Probabilities may be highly subjective and biased• Probabilities have larger uncertainties compared to

structured elicitation values




• How good are we at quantifying subjective estimates?

• Let us see…..




• How good are we at quantifying subjective estimates?– Class Example:

• How may ships passed through the Panama Canal last year?

– Give best estimate



• Background– Process developed by RAND Corporation in

late 1950’s - early 1960’s• Delphi Method• Scenario Analysis

– Effects of thermonuclear war– Civil Defense strategic planning

• Examine if U.S. population could survive a nuclear attack




• Background– Definition– A formal (protocol), heuristic (through

discussion) process of obtaining information or answers to specific questions called issues

• e.g., failure rates or probabilities, and failure consequences




• Background– EOE is used for preliminary risk evaluation

(screening) is not really intended to replace more complex reliability models

– EOE has been used by industry and government agencies to develop failure probabilities when there is a lack of failure information




• Drawbacks– Subjective process

• Not consensus building• Inherently contains bias and dominance

– Difficult to process result to determine reliability or hazard rates

• Assumptions need to be made

• Current Usage in USACE– Supplement to other models

• Calculate reliability (not for critical components)• Event tree probabilities• Used in consultation with HQUSACE




• EOE Process– Participants

• Experts• Observers• Listeners• Technical Integrator and Facilitator• Peer Reviewers

– ITR process and results




• EOE Process– Identification and Selection of Experts

• Strong relevant expertise• Familiarity and knowledge with issues• Willingness to act as impartial evaluators• Willingness to participate, prepare, and

provide needed input• Strong communication skills, interpersonal

skills, and ability to generalize




• EOE Process– Inform experts of issues

• “Read ahead” materials• Site visits

– Train experts– Elicitation

• First opinion• Discussion among experts• Second opinion




• Probability– General expressions

• Percent (1% probability of failure)• Fraction (1/100)• Relative frequency (1 out of 1000)

– Axioms of Probability

• 0 < Pf < 1

• Sum of probabilities over all possible outcomes must equal 1.

– This assume events are independent.




• Statistics– Median

• e.g., Median income, median age • Rank value• For odd n, value with rank of (n+1)/2• For even n, average of value with rank n/2 or

(n/2) + 1• Used to limit extreme values

– Average• Sum of Xi divided by sample size




Median Vs. Average

• Sample 1– 100– 100– 200– 300– 400

• Median = 200• Average = 220

• Sample 2– 100– 100– 200– 300– 2000

• Median = 200• Average = 540



• Percentiles– A p-percentile value (Xp) based on a sample is

the value of the parameter such that p% of the data is less than or equal to Xp

• e.g., The median is the 50th percentile




Chickamauga Lock and DamChickamauga Lock and Dam Expert ElicitationIssue #4 - River Wall Blocks

Event Full Description Expert-opinion elicitation SummaryName of Issue Table

First SecondQuestion #1a: Response Response

Unconstrained flow of water through the river wall adversely effecting the ability to dewater the lock and/or filling/emptying operation

Given the failure mode identified for the river wall without any advanced maintenance (fix-as-fails scenario), what is the probability of that failure by the year 2005?

Median = 0.10% Median = 0.1%

Minimum = 0.01%

Expert #1 0.0% 0.0% 25 Percentile = 0.03%Expert #2 0.0% 0.0%Expert #3 0.1% 0.1% Median = 0.10%Expert #4 0.1% 0.1%Expert #5 0.1% 0.1% 75 Percentile = 0.10%Expert #6 1.0% 1.0%

90 Percentile = 0.55%

High = 1.00%

Minimum = 0.01% 0.01%Median = 0.10% 0.10%

Maximum = 1.00% 1.00%



Chickamauga Lock and Dam Expert ElicitationIssue #4 - River Wall Filling/Emptying Cracking

Event Full Description Expert-opinion elicitation SummaryName of Issue Table

First SecondQuestion #1d: Response Response


Given the failure mode identified for the River Wall Blocks without any advanced maintenance (fix-as-fails scenario), what is the probability of that failure by the year 2050?

Median = 30.00% Median = 30.0%

Minimum = 25.00%

Expert #1 30.0% 30.0% 25 Percentile = 30.00%Expert #2 20.0% 25.0%Expert #3 40.0% 40.0% Median = 30.00%Expert #4 30.0% 30.0%Expert #5 50.0% 40.0% 75 Percentile = 37.50%Expert #6 30.0% 30.0%

90 Percentile = 40.00%

High = 40.00%

Minimum = 20.00% 25.00%Median = 30.00% 30.00%

Maximum = 50.00% 40.00%

Chickamauga Lock and Dam



Chickamauga Lock and Dam Expert ElicitationIssue #4 - River Wall Blocks

Event Full Description Expert-opinion elicitationName of Issue

First SecondQuestion #3a: Response Response


Given the fix-as-fails probability of failure for the river wall filling/emptying system, what is the probability that the chamber closure time would be 30 days for less? Between 31 and 89 days? 90 days or greater? NOTE: All three branches must add up to a total of 1.

< 30 days 31-89 days 90 day + < 30 days 31-89 days 90 day +

Expert #1 99.0% 1.0% 0.0% 99.0% 1.0% 0.0%Expert #2 80.0% 15.0% 5.0% 80.0% 15.0% 5.0%Expert #3 95.0% 5.0% 0.0% 95.0% 5.0% 0.0%Expert #4 95.0% 5.0% 0.0% 95.0% 5.0% 0.0%Expert #5 90.0% 8.0% 2.0% 90.0% 8.0% 2.0%Expert #6 85.0% 10.0% 5.0% 85.0% 10.0% 5.0%

Summary Minimum = 80.00% 1.00% 0.00% 80.00% 1.00% 0.00%Table 25 Percentile = 86.25% 5.00% 0.00% 86.25% 5.00% 0.00%

Median = 92.50% 6.50% 1.00% 92.50% 6.50% 1.00%75 Percentile = 95.00% 9.50% 4.25% 95.00% 9.50% 4.25%

Maximum = 99.00% 15.00% 5.00% 99.00% 15.00% 5.00%




Cumulative Probabality of Failure (curve-fit)

0.000

0.200

0.400

0.600

0.800

1.000

2000 2010 2020 2030 2040 2050 2060 2070

Year

Pro

bab

alit

y

Cumulative




• Class Example– Six experts required– Unknown issue given to experts

• Define assumptions of issue• Elicit first values• First results• Expert Discussion• Elicit second values• Show final elicitation results


expert-opinion elicitation

Documents