test metrics in agile - powerful tool to support changes - zavertailo iuliia
TRANSCRIPT
TEST METRICS IN AGILE TEST MANAGEMENT
A POWERFUL TOOL TO SUPPORT CHANGES
Yulia Zavertailo, Senior Test manager
03.05.2023
Agenda
− About me.
− Introduction to the client’s case.
− What needs to be improved?
− How do we visualize our results?
− A closer look at the KPIs and how to gather them.
− Conclusions.
/ 3
03.05.2023
About me
− 10 years of work experience in the field of Testing and Quality Assurance, 7.5 years in Itera.
− Moved to Oslo in November 2014. − Key competences:
− test management, − test advisory on process establishment
in a project and entire organization. − Very passionate about the ISO
9000/ISO29119 standards.− Mentor and coach for junior testers. − Love running, skiing and oil painting.
/ 4
03.05.2023
IF Skadeforsikring – general
/ 5
− Waypoint is the largest digital solution at IF built on modern technology platform with lots of interfaces for clients and internal users (1000+).
− Methodology – Agile (Scrum, Kanban).
− 13 Agile teams.
− Each team has both business and IT competence.
− IT development is done in headquarters in Oslo.
− Large Scandinavian insurance company present in the Nordic countries and Baltics with 3,6 millions of clients and 6 800 employees.
03.05.2023
– System is not covered by tests.
– No professional IT testing is done in the project.
– Business users are involved to do the functional acceptance and
regression testing.
– Long production cycle: releases are 3-4 times a year.
/ 6
IF Skadeforsikring - as a test challenge
− Critical issues found in production after release.
03.05.2023
IF Skadeforsikring - in a result
− IT test team has grown from 1 to 17 IT testers in Riga, Latvia.
− Frequency of releases is every 5 weeks.
− Response on the entire system quality and stability within a few hours.
− Project discovers and fixes 136 bugs in average every release .
− There are still functional modules which do not have enough test
coverage = facing high risk of defects.
/ 7
03.05.2023
Specifically… what are we going to improve?
1. Increase frequency of releases so that IT delivers functionality to the
business every 5 weeks by:
– Automating the regression test coverage.
– Discovering and fixing bugs earlier in the cycle.
– Improving development teams efficiency by doing continuous IT testing.
/ 8
03.05.2023
Specifically… what are we going to improve?
2. Decrease amount of bugs in the production by:
– Running automated regression suits regularly.
– Improving the quality of testing itself.
– Introducing continuous manual testing while developing.
– Focus business resources doing acceptance testing only.
/ 9
03.05.2023
Hmmm…. but how to prove the succeeded result?
Present test results our client is interested in:
– Test coverage.
– # of defects found in production.
– # of defects found in sprints.
– Time spent by end users.
– Time for feedback about system’s quality.
/ 10
03.05.2023
KPI #1 – Test Coverage
What do we measure?
− Percentage of requirements covered by at least one test.
Why do we gather it in our Agile projects?
– Visualize “black spots” of our application.
– Visualize that the larger test coverage is, the less risk of missing
defects is.
– Visualize the functionality that needs more attention since it is less
covered by tests.
/ 11
03.05.2023
Application test coverage - example
/ 12
Module 1 Module 2 Module 3 Module 4 Module 5 Module 60
200
400
600
800
1000
12001075
739
430
800
230
419
310244
130
465
70141
5413 8
694 10
Total user stories US covered manual US covered automated
03.05.2023
How to gather Test Coverage KPI
1. Configure your Agile TM tool in order to build a traceability matrix in
TM tool (TFS, Jira, QC).
2. Use the graphs module of your TM tool or export to Pivot tables in
Excel.
3. Go for the Excel-option if you do not have any TM tool.
/ 13
03.05.2023
KPI #2 - Defect open and close rates
What Do we measure?
Defect Open and Close rate = Defects found during testing in a release *100 (Defects found during testing in a release + Defects found after release)
/ 14
Why do we gather it in our Agile projects?− Categorizing defects as either open or closed.
− Certain defects could slip through the cracks and show up in the finalized
release.
− Testers and developers to work together to identify and address software
issues.
03.05.2023
Sprint vs production defects - trend example
/ 15
Releas
e 1
Releas
e 2
Releas
e 3
Releas
e 4
Releas
e 5
Releas
e 6
Releas
e 7
Releas
e 8
Releas
e 9
Releas
e 10
Releas
e 11
Releas
e 12
Releas
e 13
Releas
e 14
Releas
e 15
Releas
e 16
Releas
e 17
Releas
e 18
Releas
e 19
Releas
e 20
Releas
e 21
0
50
100
150
200
250
170
131
85 85
150
85105
129141
112
182 174152
231
175
229
139
226
175
222
124
267 13 7 10 17 10
29 20 17
47
2033
21 21 25 2413 15 18
0
144124
72 78
140
68
95 100121
95
135154
119
210
154
204
115
213
160
204
124
Total Production Defects Sprint Defects
03.05.2023
How to gather defect open and close rates KPI
To get a rate:
− Get the number of defects found during a release/sprint.
− Get the number of bugs reported by the business after release to
production.
− Calculate rate based on a formula.
To build a trend graph:
− Jira + Zephyr dashboards.
− TFS + pivot tables in Excel.
/ 16
03.05.2023
KPI #3 - Issues reported by customers
What Do we measure?
% of customer reported issues = Total # of issues reported by customers *100 total number of issues reported
Why do we gather it in our Agile projects?− Does the product meet the needs of a customer/end user?
− Effectiveness of test team.
− Types and number of defects lingering in released products.
− Critical issues with current QA processes?
/ 17
03.05.2023
Customer issues reported – example of trend
/ 18
Release 1
Release 2
Release 3
Release
4
Release
5
Releas
e 6
Releas
e 7
Release 8
Release 9
Release 10
Release 11
Release 12
Release 13
Release 14
Releas
e 15
Releas
e 16
Release 17
Release 18
Release 19
Release 20
Release 21
0
50
100
150
200
250
112 7 3 6 4 2 3 2 1 4 4 3 1 2 9 2
14 8 11 5
144124
72 78
140
68
95100
121
95
135
154
119
210
154
204
115
213
160
204
124
Customer reported isues Bugs all
03.05.2023
Issues reported by customers per functional area – example of trend
/ 19
Release1
Release2
Release3
Release4
Release5
Release6
Release7
Release8
Release9
Release10
Release11
Release12
0 5 10 15 20 25 30 35 40 45 50
10
3
10
1
6
3
9
3
2
5
15
5
6
1
3
1
1
3
2
1
1
12
2
3
1
4
4
1
5
1
3
4
3
7
2
2
1
2
1
4
6
5
5
2
5
1
2
2
2
2
1
3
9
5
5
5
4
6
4
7
6
2
2
2
1
2
4
Module1 Module2 Module3 Module4 Module5 Module6 Module7
03.05.2023
How to gather issues reported by customers KPI
To get a rate:
− Get the number of defects found by customers.
− Get the total number of defects found in a sprint.
− Calculate the rate based on a formula.
To build a trend graph:
− TFS + pivot tables in Excel.
/ 20
03.05.2023
KPI #4 – Efficiency of end users’ testing
/ 21
What do we measure?
− Time spent by end users.
− # of bugs found by end users during acceptance testing of release vs
total.
Why do we gather it in our Agile projects?
− Feedback about the quality of requirement/acceptance criteria.
− Feedback about the quality of the functional and regression testing.
− To see the cost (time spent vs amount of bugs found).
03.05.2023
Time spent by end users – example of trend
/ 22
Releas
e 1
Releas
e 2
Releas
e 3
Releas
e 4
Releas
e 5
Releas
e 6
Releas
e 7
Releas
e 8
Releas
e 9
Releas
e 10
Releas
e 11
Releas
e 12
Releas
e 13
Releas
e 14
Releas
e 15
Releas
e 16
0
50
100
150
200
250
300
350
9783
135.5
161.5
124 127 123
237
152.5135.5
288.5
112
131.5147.5
159
110
30 30 29 28.5 30 31 31 2615.5 16.5 21.5 15 14.5 16 12 13.5
80 7562
54
3447 48
40 40
12 10 7.5 6.5 8 9 5 7 6 5 8 5 10 9.5 5 6.5
30 29
79 8569 69 69
96
3948
193
54 54 54
77
42
TotalCountry Domain 1Country Domain 2Country Domain 3Country Domain 4Country Domain 5
03.05.2023
KPI #5 - Regression test suite duration
What do we measure?
− Time for feedback about system’s stability and quality – TA.
− Time for feedback about system’s stability and quality – Manual.
Why do we gather it in our Agile projects?
− Get a clue about time needed to regression-test a hot fix or feature.
− Decision-making input for further planning.
− Visualized benefit and efficiency of test automation.
/ 23
03.05.2023
Regression test suite duration - example
/ 24
Test suite # of test cases in a suite
Time to run test suite with TA, hrs
Time to run suite manually, hrs
Smoke test suite 6 0,2 1,5
Regression suite 1 191 5,5 47,75
Regression suite 2 204 5,5 51
Regression suite 3 194 5,5 48,5
Total 595 16,7 149
03.05.2023
KPIs framework – in a few simple steps
1. Decide which KPIs are important for your client (focus on KPIs 1-3).
2. Configure your test management tool and visualize.
3. Continuously use KPIs at all levels of communication.
/ 25
03.05.2023
Summary
− KPIs framework is a powerful tool to be an Agile test leader!
− Control the status of quality in a measurable way.
− Control the general healthiness of the test solution and strategy in your
project.
− Lead and make decisions in your project.
− Empower yourself to prove that IT- testing is a must in your project!
/ 26
03.05.2023
QUESTIONS?
/ 27
03.05.2023
Contacts
• LinkedIn: https://no.linkedin.com/in/yuliazavertailo • Email: [email protected]
/ 28
03.05.2023
THANK YOU!
/ 29