root cause analysis for software testers
DESCRIPTION
In many cases, we choose solutions to problems without sufficient analysis of the underlying causes. This results in implementing a cover-up of the symptoms rather than a solution to the real underlying problem. When we do this, the problem is likely to resurface in one disguise or another, and we may mishandle it again—just as we did initially. Getting to the root of the problem is the better way to solve the current problem, and save time and money in the future. Alon Linetzki identifies and explains a number of root cause analysis techniques widely used in the industry, gives examples of how to apply them in software testing, demonstrates how to implement them, and discusses how to connect them to our day-to-day testing context. Alon shares how root cause analysis can be an effective tool in defect prevention.TRANSCRIPT
![Page 1: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/1.jpg)
TP Half-day Tutorials
5/6/2014 1:00:00 PM
Root Cause Analysis for
Software Testers
Presented by:
Alon Linetzki
Best-Testing
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
![Page 2: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/2.jpg)
Alon Linetzki Best-Testing
Alon Linetzki, founder and managing director of Best-Testing, has been a coach and consultant in development, testing, and quality assurance for twenty-eight years. Alon helps organizations enhance the test engineer’s professional and personal skills, training test managers in optimization and test process improvement, and optimizing their test operations. His main areas of expertise are in agile testing and transitioning to agile, exploratory testing, test process Improvement, risk-based testing, and test automation. Alon is a member of the ISTQB® authors and review team for the ISTQB® Agile Tester Add-on certification, cofounder of ISTQB® in Israel, leader of the ISTQB® Partner Program, and founder of the SIGiST Israel conference.
![Page 3: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/3.jpg)
Root Cause Analysis in Testing
![Page 4: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/4.jpg)
Partners in this course are:
Course Partners
March 2014
2
![Page 5: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/5.jpg)
March 20143
The course materials have been prepared by:
Best – Testing,
Copyright: © Best – Testing, 2008
All materials contained in this Handbook were prepared by Best-Testing. All rights of this material are
reserved solely for Best-Testing. The material is intended for personal, noncommercial use. All
materials published in this material are protected by copyright, and owned or controlled by Best-
Testing, or the party credited as the provider of the content. You may not modify, publish, transmit,
participate in the transfer of sale of, reproduce, create new works from, distribute, perform, store on
any magnetic device or any other device, display, on in any way exploit, any of the content in whole
or in part. You may not alter or remove any trademark, copyright or other notice from copies of the
content. You may no use the material in this material for the purpose of training of any kind, internal
or for customers, without beforehand a written approval of Best-Testing.
The Use of this material
The material in this material is designed to assist the student during the course. It does not include all
of the information that will be referred to during the course and should not be regarded as a
replacement for reference manuals or other instructions.
Copyrights
![Page 6: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/6.jpg)
March 20144
Limit of responsibility
Best-Testing invests significant effort in updating this material, however, Best-Testing is not responsible of
any errors or material which may not meet specific requirements. The user alone is responsible for
decisions based on the information contained in this material.
Protected Trademark
In this material, protected trademarks appear that are under copyright. All rights to the trademarks in this
material are reserved to the authors.
Best–Testing wishes you success in the course!
Copyrights
![Page 7: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/7.jpg)
Introduction
March 2014
5
• No bullets…should be fun…
Chapter 1
![Page 8: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/8.jpg)
6
Alon Linetzki
30 years in IT, 20+ years as a test engineer and a test manager
Certified Scrum Master, Scrum Alliance, 2008
ISQTB® Partner Program leader, ISTQB® Agile Tester Certification Leader
Specializes in Agile testing test strategy & optimizationtest process improvement test management test design risk based testing test automation Building Smart Teams
A degree in ‘testing’ – B.Sc. statistics and criminology,
International Speaker worldwide, since 1995
Co-founder of the Israeli Testing Certification Board (www.itcb.org.il, 2004)
Founder of the SIGiST Israel (testing forum) in Israel (www.sigist.org.il, 2000)
![Page 9: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/9.jpg)
We shall cover…
Introduction
Presenting participants and trainer
Workshop expectations
What is Root Cause Analysis?
How to use RCA for Defect analysis?
7
March 2014
![Page 10: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/10.jpg)
We shall cover…
How to Use RCA for analyzing critical problems?
Cause Effect graphing, amended
Case Study #0
Summary
How to use RCA for process improvement?
Case Study #1
Case Study #2
Retrospective
Workshop retrospective
8
March 2014
(*) Bonus topic – only if there is enough time left…
![Page 11: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/11.jpg)
Professional Background
Years experience in testing/development/both/other
Responsibilities in current role
Have you implemented RCA? Can you share?
Workshop expectations?
Introduce participants…
9
March 2014
![Page 12: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/12.jpg)
Interactive sessions
Discussions, skepticism, curiosity
Examples, demonstration
Thinking, communicating
Keeping an open mind for ideas and concepts
Contribute to the class team
Learn…discuss…review…
Give quick feedback on everything you may think of
Time boxed sessions…breaks…exercises…discussions
Introducing workshop dynamics10
March 2014
![Page 13: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/13.jpg)
There are FAR more slides than we can
cover…
Say if you have something to say… do not
wait till the end of the workshop!
Introducing workshop dynamics11
March 2014
![Page 14: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/14.jpg)
Start : 10:15
Lunch: 11:45 – 12:45
Finish line: 14:15
But… we may have more
breaks if we feel like it…
Logistics…
March 2014
12
![Page 15: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/15.jpg)
What Root Cause Analysis?
March 2014
13Chapter 2
![Page 16: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/16.jpg)
Root Cause Analysis definition(My interpretation)
From wiki:
Root cause analysis (RCA) is a class of problem solving
methods aimed at identifying the root causes of problems
or events.
The practice of RCA is predicated on the belief that
problems are best solved by attempting to correct or
eliminate root causes, as opposed to merely addressing
the immediately obvious symptoms.
By directing corrective measures at root causes, it is
hoped that the likelihood of problem recurrence will be
minimized.
14
![Page 17: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/17.jpg)
Root Cause Analysis definition
From Bill Willson’s website:
Root cause analysis (RCA) is a methodology for
finding and correcting the most important reasons
for performance problems.
It differs from troubleshooting and problem-solving
in that these disciplines typically seek solutions to specific difficulties, whereas RCA is directed at
underlying issues
15
![Page 18: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/18.jpg)
Root Cause Analysis - definition
Root cause analysis (RCA) is a methodology for finding
and correcting the most important reasons for
performance problems.
It differs from troubleshooting and problem-solving in
that these disciplines typically seek solutions to
specific difficulties, whereas RCA is directed at
underlying issues.
16
![Page 19: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/19.jpg)
Root Cause Analysis definition -summary
As a business process improvement tool, RCA seeks
out unnecessary constraints as well as inadequate
controls.
In safety and risk management, it looks for both
unrecognized hazards and broken or missing barriers.
It helps target CAPA (corrective action and preventive
action) efforts at the points of most leverage.
RCA is an essential ingredient in pointing
organizational change efforts in the right direction.
Finally, it is probably the only way to find the core
issues contributing to your toughest problems.
17
![Page 20: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/20.jpg)
Root Cause Analysis(My interpretation)
A problem solving method/process designed to search for the root causes of a problem using a predefined structural thinking process, identifying the underlying issues, with the expectation that dealing with these issues will dramatically reduce the likelihood of the problem to occur.
The process involves data collection, cause charting, root cause identification and recommendation generation and implementation
18
![Page 21: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/21.jpg)
“Cows falling on a road from a mountain”
– is it a problem or a symptom?
Should we eliminate all cows on that area?
Should we dig-out the mountain?
Should we rotate the sign?
Should we divert the road elsewhere?
It seems that sometimes eliminating the causes is not an easy task, and finding the problems is even harder!
19
![Page 22: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/22.jpg)
Root causes are underlying causes
The investigator’s goal should be to identify
specific underlying causes
The more specific the investigator can be about
why an event occurred, the easier it will be to
arrive at recommendations that will prevent
recurrence.
Practical RCA – definition guidelines
March 2014
20
![Page 23: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/23.jpg)
Root causes are those that can reasonably be
identified
Occurrence investigations must be cost beneficial
It is not practical to keep valuable manpower
occupied indefinitely searching for the root
causes of occurrences…
Structured RCA helps analysts get the most out of
the time they have invested in the investigation.
Practical RCA – definition guidelines
March 2014
21
![Page 24: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/24.jpg)
Root causes are those over which management has control
Analysts should avoid using general cause classifications such as “operator error”, “equipment failure” or “external factor”
Such causes are not specific enough to allow management to make effective changes
Management needs to know exactly why a failure occurred before action can be taken to prevent recurrence
We must also identify a root cause that management can influence:
Identifying “severe weather” as the root cause of parts not being delivered on time to customers is not appropriate
Severe weather is not controlled by management.
Practical RCA – definition guidelines
March 2014
22
![Page 25: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/25.jpg)
Root causes are those for which effective recommendations can
be generated (if cannot be done, keep on searching!)
Recommendations should directly address the root causes
identified during the investigation
If the analysts arrive at vague recommendations such as,
“Improve adherence to written policies and procedures,” then
they probably have not found a basic and specific enough
cause and need to expend more effort in the analysis process.
Practical RCA – definition guidelines
March 2014
23
![Page 26: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/26.jpg)
Root causes are underlying causes
Root causes are those that can reasonably be
identified
Root causes are those over which
management has control
Root causes are those for which effective
recommendations can be generated (if
cannot be done, keep on searching!)
Practical RCA – definition guidelines
March 2014
24
![Page 27: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/27.jpg)
Time for discussion…25
March 2014
![Page 28: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/28.jpg)
Root Cause Analysis for Beginners - by James J. Rooney and Lee N. Vanden Heuvel
References – what is Root Cause Analysis?26
March 2014
![Page 29: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/29.jpg)
How to use RCA in Defect Analysis?
March 2014
27
![Page 30: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/30.jpg)
Escaping defects occur all the time
We try to contain them, from going to production,
but much efforts should be done in preventing
them going from Design to Code
A set of questions may help out leading us to Data
Collection on those Defects for performing the
RCA – next page…
Defects RCA introduction
March 2014
28
![Page 31: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/31.jpg)
Defects RCA guided questions
Data Collection
March 2014
29
Question Who
asks?
Describe circumstances and consequence of the
Problem?
Test Team
Where was the problem introduced? Dev Team
Describe the root cause of the problem Dev Team
What reviews were held for the phase? Dev Team
How and why did the problem escape testing? Test Team
What could be done to prevent this type of error in the
future?
Both
What could be done to find this type of problem in the
future?
Both
John Ruberto – Root Cause for Problems Escape
![Page 32: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/32.jpg)
Defect Analysis is the process of analyzing a defect
to determine its root cause.
Defect Prevention is the process of addressing
root causes of defects to prevent their future
occurrence.
RCA – few definitions
March 2014
30
![Page 33: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/33.jpg)
Defect Source – Finding defects in the stage they
were introduced and as early in the lifecycle as
possible – Eliminating escaping defects
RCA – few definitions
March 2014
31
![Page 34: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/34.jpg)
Defects Root Cause Analysis 32
![Page 35: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/35.jpg)
Canceled Defects Root Cause Analysis
Cancelled defects are not real defects of the
system-under-test
They can be the result of: environment problem
(non product), out of scope, test design or test
execution problem, un-reproducible defect, not
understanding the specifications, etc.
33
![Page 36: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/36.jpg)
Canceled Defects Root Cause Analysis34
![Page 37: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/37.jpg)
Code Inspection - Example
March 2014
35
Background:
180 people on the project
3,700 modules interconnecting with each other
Complex infrastructure
25 testers
Using Test Director/Quality Center as reporting
platform
Strategic customer
![Page 38: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/38.jpg)
Code Inspection - Example
March 2014
36
![Page 39: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/39.jpg)
Code Inspection - Example
March 2014
37
Analysis:
Main concern – Complience with design (10%
Medium+)
Investigated RCA:
A few specs were written by same designer
A few modules were written by 3 programmers
Initiated training for designer and programmers
![Page 40: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/40.jpg)
Code Inspection - Example
March 2014
38
Background:
Another project,
450 people
working, 55 testers,
4,300 modules,
strategic customer
Focus on:
Performance
Maintainability
Proper algorithm
usage
![Page 41: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/41.jpg)
Code Inspection - Example
March 2014
39
Analysis:
Investigated RCA:
Performance defects were on resource usage, memory
leaks
Implementing improper algorithm was mainly due to
specification documents written unclear + implementation
complexity chosen by a few programmers
Initiated training for architects and programmers
![Page 42: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/42.jpg)
Adding defect template fields:
Injected, Detected, Removed
Version [build] + Date per field
[Updating the defect lifecycle process (status)]
Changing field CRUD permissions – Dev, Test, Managers
Defining reports
Updating the Defect Management process
Training the teams
Implementing action plan…
Implementation guidelines –
defects RCA
March 2014
40
![Page 43: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/43.jpg)
RCA process defined
March 2014
41
![Page 44: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/44.jpg)
Time for discussion…42
March 2014
![Page 45: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/45.jpg)
John Ruberto – Root Cause for Problems Escape,
http://blog.ruberto.com/2008/04/another-post/
References
March 2014
43
![Page 46: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/46.jpg)
How to Use RCA for analyzing critical
problems?
March 2014
44
![Page 47: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/47.jpg)
Challenges the current method could
not solve – using 5whys
45
![Page 48: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/48.jpg)
‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram.
Presenting a problem,
Asking “why?” it happens, finding the effect that caused it
(1 effect),
Presenting the effect on the diagram,
Asking “why?” it happens… [back to previous step, unless
we ask it for 5 times already]
Done.
Presenting the 5whys Technique46
![Page 49: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/49.jpg)
‘5ys’ or ‘5 whys’ technique, and the cause-effect
diagram.
Presenting the RCA Technique47
CauseCause Cause Cause
Cause
Problem
Why
#1
Why
#2Why
#3Why
#4
Why
#5
Thinking path…
![Page 50: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/50.jpg)
‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram.
1. There is the assumption that a single cause, at each level
of "why", is sufficient to explain the effect in question.
2. What if one of the ‘Why’ is answered wrongly? Maybe
our answer is possible, but what if the actual cause is
something else entirely?
3. When we have found the problem, and draw the route,
how ‘strong’ is this solution? Maybe we should prefer one
over the other?
Challenges: what the method can
not solve48
![Page 51: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/51.jpg)
Enhancing the method – case study49
![Page 52: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/52.jpg)
Short structured interview with rep’s of management, development, release, system, testing, product teams.
Step 1: Draw a cause-effect diagram & exercise the 5whys
Step 2: Investigate the arrows/causes for:
What evidence you have that the cause exist? (relevancy)
What evidence you have that the cause leads to the effect? (strength)
Is anything else needed together with the cause for the effect to occur? (strength)
Is there evidence that the cause is contributing to the problem I’m looking at? How much it contributes? (Impact: direct/indirect)
Enhancing the Method:
Example project50
![Page 53: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/53.jpg)
Enhancing the Method:
Example project51
Type Question Cause-Effect
Relevancy What evidence you have that the cause exist? H/M/L
Strength
(S or W)
What evidence you have that the cause leads to
the effect?
H/M/L
Strength
(S or W)
Is anything else needed, together with the
cause, for the effect to occur?
Yes/No
Impact
(D or I)
Is there a evidence that the cause is contributing
to the problem I’m looking at?
Yes / No
Impact
(D or I)
How much this cause is contributing to a
possible resolution?
Direct /
Indirect
Mark
![Page 54: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/54.jpg)
Enhancing the Method:
Example project52
Type Question Cause-Effect
Relevancy What evidence you have that the cause exist? High (3)
Strength
(S or W)
What evidence you have that the cause leads to
the effect?
Medium (2)
Strength
(S or W)
Is anything else needed, together with the
cause, for the effect to occur?
Yes (1)
Impact
(D or I)
Is there a evidence that the cause is contributing
to the problem I’m looking at?
Yes (1)
Impact How much this cause is contributing to a
possible resolution?
Direct (2)
Mark 9
You should mark each arrow using this table.
![Page 55: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/55.jpg)
Step 3: Identify the routes leading to the
problem/s,
Step 4: Identify the strength and direction (impact)
they have (calculating the mark for each arrow),
Step 5: Choose the best route to focus on,
[Improve it, and go to the next one].
Enhancing the Method:
Example project53
![Page 56: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/56.jpg)
Case Study - implementation54
![Page 57: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/57.jpg)
Background:
company was using a very advanced technology, and
a complex product line,
Complex product, uses mechanics, electronics,
hardware, software, devices, cooling device, has water
resistant, has heating resistant, accurate up to
1:1,000,000 cm,
In the last 0.5 year, 50% of released machines
returned from the floor (clients) for fixing,
Example project – Hi-Tech Company55
![Page 58: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/58.jpg)
SQA manager was at a course I gave, and liked
one of the tools,
He thought automation can solve many of his
problems, because:
A lot more tests running,
Identifying more defects before the clients do,
Less products coming back,
Clients are happy!
Example project – Hi-Tech Company56
![Page 59: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/59.jpg)
I investigated their automation
needs,
Followed the steps of the
enhanced method,
Found out their problems might
be elsewhere…
Example project – Hi-Tech Company57
Lets see the drawing board from
that meeting…
![Page 60: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/60.jpg)
1st drawing – RCA meeting
58
Our way of thinking12
![Page 61: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/61.jpg)
The RCA meeting (company exec’s and directors):
At first, the belief was that the primary problem
was:
Partial Test Planning (less tests are executed)
Example project – Hi-Tech Company59
Lets see an illustration diagram …
![Page 62: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/62.jpg)
1st drawing – RCA meeting60
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of
version control
management is
very high
Defining req’
not good
enough by
client
Spec Lvl 0
No specs
in lvl 1
Spec Lvl
1 not
complete
or does
not fit
Spec Lvl
2 not
written
Good
definition of
Spec Lvl 0
Spec Lvl
1 fits
Spec Lvl
2 fit
Spec Lvl 2
does not
fit/complete
Code
written
with low
match to
client req’
Only
Partial Test
planning
and not full
coverage
Partial test
case planning
and coverage
Partial test
execution
and low
coverage
Our way of thinking1
2
![Page 63: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/63.jpg)
1st drawing – RCA meeting61
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of
version control
management is
very high
Defining req’
not good
enough by
client
Spec Lvl 0
No specs
in lvl 1
Spec Lvl
1 not
complete
or does
not fit
Spec Lvl
2 not
written
Good
definition of
Spec Lvl 0
Spec Lvl
1 fits
Spec Lvl
2 fit
Spec Lvl 2
does not
fit/complete
Code
written
with low
match to
client req’
Only
Partial Test
planning
and not full
coverage
Partial test
case planning
and coverage
Partial test
execution
and low
coverage
Our way of thinking1
2
![Page 64: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/64.jpg)
1st drawing – RCA meeting62
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of
version control
management is
very high
Defining req’
not good
enough by
client
Spec Lvl 0
No specs
in lvl 1
Spec Lvl
1 not
complete
or does
not fit
Spec Lvl
2 not
written
Good
definition of
Spec Lvl 0
Spec Lvl
1 fits
Spec Lvl
2 fit
Spec Lvl 2
does not
fit/complete
Code
written
with low
match to
client req’
Only
Partial Test
planning
and not full
coverage
Partial test
case planning
and coverage
Partial test
execution
and low
coverage
Our way of thinking1
2
![Page 65: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/65.jpg)
2nd drawing – RCA meeting63
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
Complexity of
version control
management is
very high
Defining req’
not good
enough by
client
Spec Lvl 0
No specs
in lvl 1
Spec Lvl
1 not
complete
or does
not fit
Spec Lvl
2 not
written
Good
definition of
Spec Lvl 0
Spec Lvl
1 fits
Spec Lvl
2 fit
Spec Lvl 2
does not
fit/complete
Code
written
with low
match to
client req’
Only
Partial Test
planning
and not full
coverage
Partial test
case planning
and coverage
Partial test
execution
and low
coverage
Our way of thinking1 2
![Page 66: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/66.jpg)
After a while, we shifted the focus and agreed that
the real problem was actually:
Poor Product Quality
Because that was the reason the clients returned
their product.
And we started RCA from there.
After a while, we started to see the light – real
problems started to crystallize, problems that
involved people and processes
Example project – Hi-Tech Company64
![Page 67: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/67.jpg)
3rd drawing – RCA meeting
65
![Page 68: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/68.jpg)
Our way of thinking
3rd drawing – RCA meeting66
Many clients
ask for
different Sw
of the
product
Many
versions
open in
parallel
SCM - Complexity
of version control
management is very
high
Defining req’ not
good enough by
client
Spec Lvl 0
No
specs in
lvl 1
Spec Lvl 1
not complete
or does not
fit
Spec Lvl
2 not
written
Good
definition
of Spec
Lvl 0
Spec Lvl
1 fits
Spec Lvl
2 fit
Spec Lvl 2
does not
fit/complete
Code written
with low
match to
client req’
Only
Partial Test
planning
and not full
coverage
Partial test
case
planning and
coverage
Partial test
execution
and low
coverage
1 2
Tight
schedule
projectPrioritization
and
compromise
on scope to
clients
Low
Quality
Product
Req’
managemen
t not good
enoughLack of methods
and techniques
in testing
Low lvl of test
identification
![Page 69: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/69.jpg)
We then defined the relevancy, strength and
impact of each arrow (cause),
And calculated the grades for the arrows (which
are not seen here),
Example project – Hi-Tech Company67
Back to the board…
![Page 70: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/70.jpg)
4th drawing – RCA meeting
68
![Page 71: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/71.jpg)
5th drawing – RCA meeting69
Many
clients ask
for
different
Sw of the
product
Many
versions
open in
parallel
SCM - Complexity of version control
management is very high
Defining
req’ not
good
enough by
client
Spec Lvl 0
No specs in
lvl 1
Spec Lvl 1
not
complete
or does not
fit
Spec Lvl 2
not written
Good
definition
of Spec Lvl
0
Spec Lvl 1
fits
Spec Lvl 2
fit
Spec Lvl 2
does not
fit/complete
Code
written
with low
match to
client req’
Only
Partial Test
planning
and not
full
coverage
Partial test
case
planning
and
coverage
Partial test
execution
and low
coverage
Our way of thinking12
Tight
schedule
project
Prioritization
and
compromise on
scope to clients
Low
Quality
Product
Req’
management
not good
enoughLack of methods and
techniques in testing
Low lvl of test
identification
S/D
W/D
W/I
S/I
![Page 72: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/72.jpg)
We went back to double check the RCA of the
routes leading to the primary problem, marking
the arrows with their grades (from the table,
remember?)
We ended up circling the main causes, that have
initiated the strongest routes that are directly
impacting our problem,
Example project – Hi-Tech Company70
![Page 73: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/73.jpg)
6th drawing – RCA meeting
71
![Page 74: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/74.jpg)
Last drawing – RCA meeting72
![Page 75: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/75.jpg)
Many
clients ask
for
different
Sw of the
product
Many
versions
open in
parallel
SCM - Complexity of version control
management is very high
Defining
req’ not
good
enough by
client
Spec Lvl 0
No specs in
lvl 1
Spec Lvl 1
not
complete
or does not
fit
Spec Lvl 2
not written
Good
definition
of Spec Lvl
0
Spec Lvl 1
fits
Spec Lvl 2
fit
Spec Lvl 2
does not
fit/complete
Code
written
with low
match to
client req’
Only
Partial Test
planning
and not
full
coverage
Partial test
case
planning
and
coverage
Partial test
execution
and low
coverage
Our way of thinking1
2
Tight
schedule
project
Prioritization
and
compromise on
scope to clients
Low
Quality
Product
Req’
manageme
nt not
good
enoughLack of methods and
techniques in testing
Low lvl of test
identification
S/D
W/D
W/I
S/I
Last drawing – RCA meeting73
![Page 76: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/76.jpg)
Many
clients ask
for
different
Sw of the
product
Many
versions
open in
parallel
SCM - Complexity of version control
management is very high
Defining
req’ not
good
enough by
client
Spec Lvl 0
No specs in
lvl 1
Spec Lvl 1
not
complete
or does not
fit
Spec Lvl 2
not written
Good
definition
of Spec Lvl
0
Spec Lvl 1
fits
Spec Lvl 2
fit
Spec Lvl 2
does not
fit/complete
Code
written
with low
match to
client req’
Only
Partial Test
planning
and not
full
coverage
Partial test
case
planning
and
coverage
Partial test
execution
and low
coverage
Our way of thinking1
2
Tight
schedule
project
Prioritization
and
compromise on
scope to clients
Low
Quality
Product
Req’
manageme
nt not
good
enoughLack of methods and
techniques in testing
Low lvl of test
identification
S/D
W/D
W/I
S/I
74
4/6
Last drawing – RCA meetingLets see the routes…
3/3
![Page 77: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/77.jpg)
5 major Root Topics were Identified, explained and prioritized:
1. Produce requirements from client definitions
2. Requirements management
3. Either ‘No Spec Level 1’, or ‘Spec level 1 not matching requirements’
4. Lack of methods and techniques in testing for development and testing teams
5. Allot of clients define slightly different requirement for the SW – allot of specials
We defined a pragmatic corrective actions plan, with priority items.
Example project – Hi-Tech Company75
![Page 78: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/78.jpg)
Major Areas of Concern identified and prioritized:
1. Requirements Management
2. Configuration Management
3. Design Documentation and Flow
4. Testing Methodologies, techniques and tools
Not discussed:
- Release Management
- Risk Management + Risk Based Testing
- Requirements Definition
- Project Management
- Professional Development
Example project – Hi-Tech
Company76
Organization Language!
![Page 79: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/79.jpg)
Differentiating problems from
symptoms
77
![Page 80: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/80.jpg)
If we follow the questioning and grading process, than problems shall be the ‘leafs’ of the cause-effect diagram, with:
A positive answer to the questions:
Relevancy – ‘yes’
Highest strength mark (direct/Indirect), and
Direct impact on the undesired outcome (initial hypothesis)
Highest grade per route guideline,
Translate into company language (areas of concern)
Enables an Immediate vs. long term action plan
Differentiating Problems from
Symptoms78
Just what our office needs..
![Page 81: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/81.jpg)
Solving problems rather than
covering symptoms
79
![Page 82: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/82.jpg)
Cause-effect & RCA may eliminate the problems of the model
when answering the questions and marking them on the diagram
for the best route(s),
(but) We must deal with symptoms some time, in the short term,
We should estimate effort needed (or money required) for each
route, and integrate that into our decision (i.e. 60/40 weight)
The questions (answers) make sure that our analysis is with
minimal deviations, and that the route we take is a ‘strongest’ one,
It is a constructive process that leads to Understanding, Clarity,
Focus and the right Priority setting
Summary 80
![Page 83: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/83.jpg)
Further enhancing the mode, we must think of the following:
What about the junctions points (inbound and outbound):
direct impact of routes with those? Indirect? Impact on speed of
performance (bottle-necks)?
What is the ROI of this method within context?
Can we validate a route? Can we tie it to be a successful
problem eliminator?
How much the method is context dependant?
Can we hook it to Test Process Improvement methods or other
Key Performance/Area Indicators?
Other?
Food for Thought…81
![Page 84: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/84.jpg)
Time for discussion…82
March 2014
![Page 85: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/85.jpg)
RCA and Process Improvement?
March 2014
83
Introduction
Case Study #1
Case Study #2
![Page 86: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/86.jpg)
Process improvement should use data and RCA
outputs
CI example… - process was changed, training was
done to relevant people
Defects example… - lessons learned, training was done
to relevant testers, discussions with development
made it clear which information is required more on
the defect template
RCA and Process Improvement?
March 2014
84
![Page 87: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/87.jpg)
Analyzing test environments
utilization downtime
March 2014
85
![Page 88: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/88.jpg)
Analysis:
Small, yet impacting, breaks in work, caused huge
impact on the release (big team of testers)
Breaks were 15-45 min each
Areas of impact were: performance, network, DB
Analyzing test environments
utilization downtime
March 2014
86
![Page 89: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/89.jpg)
Improved process:
Defined measurements for monitoring downtime
of test environments
Identify trend of areas of impact on infrastructure
Used it weekly to establish Status of Downtime, to
decrease technical impacts (reached -25% waste
on next release).
Analyzing test environments
utilization downtime
March 2014
87
![Page 90: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/90.jpg)
Analyzing development effort
with defects found
March 2014
88
15
26
11
120
65
43
32
187
12
38
210
294
588
2184
504
210
126
1344
336378
80%
99%
21%
62%
145%
230%
286%
156%
40%
113%
0%
50%
100%
150%
200%
250%
300%
1
50
SC MAF FBF CSM E-Care E-Support Rater AR Reports BL Interface
Investment vs Defects
Defects Dev Effort Def % vs Dev Eff %
~28 years development.
~150 people in development
team.
![Page 91: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/91.jpg)
Analysis:
Areas of defects clustering mainly in: Rater, E-Support, E-Care, AR
Relative % of defects are much higher than % of development effort (days)
Investigated UT and Staging (integration test) phases, found missing testing types done, UT & INT regression testing, performance testing, interface testing
Further considerations: Complexity, New/Legacy code, # people touching code
Analyzing development effort
with defects found
March 2014
89
![Page 92: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/92.jpg)
Improved process:
Defined Unit test and Integration test process and
strategy guidelines and work instructions
Trained programmers in UT and INT techniques and
new processes
Aligned effort estimation and duration with project
management and product management
Defined coverage criteria for UT and INT
Initiated test automation project on UT and INT levels
Analyzing development effort
with defects found
March 2014
90
![Page 93: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/93.jpg)
Time for discussion…91
March 2014
![Page 94: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/94.jpg)
Root Cause Analysis techniques can help us in
finding the underlying problems (root causes), and
get to deal with real problems
Structured RCA methods support productive
thinking, identifying problems and identify more
symptoms
Cause Effect Graphing (amended), Defects RCA
RCA and process improvement
Summary
Retrospective
March 2014
92
![Page 95: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/95.jpg)
“It is not the strongest of the species that survives, nor the most intelligent but the one that is most responsive to change”
Charles Darwin
A changing world…93
![Page 96: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/96.jpg)
Or perhaps . . .
94
. . . the one who had anticipated all possible
requirements !
![Page 97: Root Cause Analysis for Software Testers](https://reader034.vdocuments.site/reader034/viewer/2022051013/5482ee9fb47959e70c8b4955/html5/thumbnails/97.jpg)
Root Cause Analysis in Testing