2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation

June 23rd , 2010

Presenter: Terry J. Herrmann, P.E.

Associate, Structural Integrity Associates

therrmann@structint.com

16th Annual HPRCT ConferenceJune 21-24 · Sheraton Inner Harbor HotelBaltimore, MDHosted by Constellation Energy

Completing a High Quality Root Cause in 3-Weeks or

Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation

Your Presenter

• BS Mechanical Engineering & MS Engineering Management from Syracuse University• Over 30 years experience in power generation in the areas of design, construction, testing, failure / root cause analysis, equipment reliability, and probabilistic risk assessment.• Developed and implemented programs in root cause analysis, system engineering, and risk-based applications.• Recipient of 2002 Kepner-Tregoe® International Rational Process Achievement Award.• IEEE Subcommittee on Human Factors, Control Facilities and Human Reliability – Recommended Practice for Investigation of Events at Nuclear Power Plants. • Contributor to EPRI Report 1016907, Preservation of Failed Parts to Facilitate Failure Analysis of Nuclear Power Plant Components

Terry J. Herrmann, P.E.

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation

Completing a High Quality Root Cause in 3-Weeks or Less

3

Participant Input for this Presentation:

• Provide one key factor that affected your ability to complete an assigned root cause evaluation within the allotted time.

• Provide one brief example of a success.• Name one or two key “take-aways” you are most interested

in getting from this presentation.

2010 HPRCT Presentation – Optimized Human Error Evaluation

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 4

The objective of performing a Root Cause Analysis is to optimize the use of the

organizations’ resources (time and cost) in achieving an effective, long-lasting solution to

identified problems.

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 5

Presentation Outline:

• Factors that affect RCA quality.• Evidence preservation.• Organizational support.• Lead investigator experience.• Planning & Organization.• Learning from your own experiences.• Pitfalls to avoid.• Topics for Discussion.

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 6

Factors that Affect RCA Quality & Timeliness:

Evidence Preservation Effectiveness.

Organizational Support for the Investigation.• Clear expectations.• Sufficient resources.• Sufficient independence.• Progress monitored.

Experience of the RCA lead investigator.

Planning.

What else?

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 7

Evidence Preservation Effectiveness:

1.Have a pre-defined plan for evidence preservation.

2.Preserve the failure location and start collecting information as soon as possible after the failure.

3.Collect information from people in the vicinity and from those who were operating the equipment - before the end of their work shift, if possible.

4.Take photographs to show the general arrangement from

a distance and close-ups of the parts of interest.

5.Obtain records that are volatile (e.g., post-trip logs) before they disappear.How well prepared are you for rapid evidence collection?

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 8

Organizational Support:

• Clearly defined management expectations.• For initiating the investigation.• For key milestones (e.g., RCA team assigned, investigation charter approved, draft report date, etc.)• For management/RCA team roles/responsibilities.

• Resources available to support the investigation.• RCA team.• Other company experts.• Laboratory / consultants for supporting analyses.

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 9

Organizational Support:• Sufficient independence.

• For leaving no relevant stone unturned.• For being able to report “unpopular” results without fear of retaliation or being told to re-work the RCA.• Through independent third-party reviews.

• Progress is regularly monitored.• To make sure the RCA team is getting needed support.• To ensure the team remains focused on completing the RCA on time and isn’t diverted to other work.

Do you typically have the level of support you need?

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 10

RCA Lead Investigator Experience:• Lead investigator should have considerable experience in performing significant event investigations.

• Qualification process should ensure this.• If experience is lacking, need to obtain help rather than trying to gain it on the job.• Fewer, high qualified people much better than more less qualified individuals.

How many people in your organization have a significant level of experience leading an RCA team?

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 11

RCA Planning:• A RCA should be managed as a project. It has:

• Activities.• Timelines.• Milestones.• Project deliverables.

• This is impacted by:• Availability of inputs (evidence preservation & collection)• Resource constraints (people, budget, etc.).

Completing a High Quality Root Cause in 3-Weeks or Less

How well do you manage your RCA projects?

What areas need the most improvement?

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 12

Learning from your own experiences:• Every RCA should have a pre-job brief with all team members present to:

• Review expectations.• Distribute tasks.• Establish monitoring points.• What else?

Completing a High Quality Root Cause in 3-Weeks or Less

Do you find this to be helpful? Why or why not?

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 13

Learning from your own experiences:• Every RCA should perform a post-CARB critique:

• With all team members.

• With people having key roles, such as:

• Assigned “owner” of the RCA.

• CARB members.

• Independent reviewer.

• RCA program owner (if you have one).

• Consultants (if used).

• Identify what went well and what didn’t.

• Collect this information and periodically review it for items needing improvement.

Completing a High Quality Root Cause in 3-Weeks or Less

Do you do this? Why or why not?

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 14

Pitfalls to Avoid:

• Thinking that taking longer makes for higher quality:• Starting the investigation late reduces quality.

• Not having help available when needed reduces quality.

• More often, schedule slips are due to a lack of focused efforts on the part of the team, which reduces quality.

• Not allowing for reasonable delays when it leads to improved quality (overly rigid timelines):

• The complexity of lab or engineering analysis may warrant additional time.

• Collecting validating field information may require specific plant conditions only available at certain times.

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 15

Pitfalls to Avoid:

• Not getting a critical independent review:• Nothing says we “have to”.

• Weak reviewers:

• People overly concerned about looking bad in front of management, the regulator, etc.

• Thinking you don’t need help when you actually do:

• This causes delays once you realize you’re not getting to the cause.

• Needs to be a check early in the process for this.

Completing a High Quality Root Cause in 3-Weeks or Less

2010 HPRCT Presentation – Optimized Human Error Evaluation2010 HPRCT Presentation – Optimized Human Error Evaluation 16

Topics for Discussion:

• What are your biggest challenges for:

• Getting the quality you want?

• Getting the RCA done timely?

• What have you found works well?

• Other items you’d like to discuss..

Completing a High Quality Root Cause in 3-Weeks or Less