tricentis webinar: mastering your business risk
TRANSCRIPT
© 2016 by© 2016 by .
Mastering your Business RiskElmar Pauwels
© 2016 by
Acceptance
Deployment
Waterfall
Requirements
Design
Implementation
Testing
Oops, Hello Mr. Bugs…!
eternally long∞
delivery cycle time𝟎
Evolutionary Testing Methodologies
© 2016 by
∞ 𝟎shortish
DEV DEV
OPS OPS
DEV DEV DEV
It compiles, it works on my machine and therefore it works!
a little less a lot lesseternally long
delivery cycle time
OPS → OOPS!
Agile
Siloisation
Evolutionary Testing Methodologies
© 2016 by
DevOps
∞ 𝟎close to zero
Dev Ops
ContinuousRelease & DeploymentCollaborative
Development
ContinuousTesting
ContinuousMonitoring
ContinuousIntegration
ContinuousFeedback
Today!
Tomorrow!?
InsaneSpeed
SpookyAction
shortisha little less a lot lesseternally long
delivery cycle time
…our conclusion
…his conclusion
Evolutionary Testing Methodologies
© 2016 by
Enterprise system landscapes are alike disease gene networks
ContinuousTesting
∞ 𝟎close to zero
InsaneSpeed
SpookyAction
shortisha little less a lot lesseternally long
delivery cycle time
…our conclusion
…his conclusion
HolyTestingGrail
?Complexity Quantity VarietySpeed
Today!
Tomorrow!?
Challenges
It’s not equal to automation!
It is about
How to keep the pace?
© 2016 by
As a tester I strive after continuous testing in order to provide instant feedback about the quality of the
product at any time.*Being able to execute the maximum amount of the most
important test cases automatically on demand to remove as much as possible business risk earliest possible.
What is continuous testing?
© 2016 by
Now
Then
Test
Cas
es
Auto
mat
ion
Leve
l
Risk Coverage
High GoalOriented
ValueOriented
QualityOriented
ProcessOriented
TimeOriented
EffortOrientedHigh
Low
Low
Continuous Testing
Overall Goal
© 2016 by
Leave your shoesbehind and drive.
Automate
A
B
ManageKeep the traffic
lights green.
1
Why isn’t testing just about speed?
OptimizeFind the shortest
possible path.
right
right
do things
do the
things
Effectivity
Efficiency
GoalOriented
ValueOriented
QualityOriented
ProcessOriented
TimeOriented
EffortOriented
© 2016 by
Passed
Failed
Not Executed
Leak
Business Risk
Passed
Failed
Not Executed
Test Cases
Measuring quality in terms of risk is to know, i.e. if you can‘t measure it, you can‘t improve it.
scrutinizing test reportsBad Good
From Counting Test Cases to Risk Coverage
© 2016 by
Risk quantifies the potential oflosing something of value.
*That’s our axiom. A premise or starting point of reasoning. It’s a premise so evident as to be accepted as true without
controversy.
© 2016 by
Securities TradingCapture Order
Client Side Validation
Market Side Validation
Check EligibilityCheck SuitabilityCheck Availability
Rectify OrderCancel Order
256
AbsoluteWeight
512
1024
64
64
1024
32
32
4 4
5 4
5 5
3 3
3 3
5 5
2 3
1 4
DamageClass
FrequencyClass
It‘s all about relative comparisons!
© 2016 by
256
AbsoluteWeight
512
1024
64
64
1024
32
32
80% 80%
26.7% 33.3%
23.7% 88.8%
1.5% 5.6%
1.5% 5.6%
53.3% 66.7%
10% 10%
10% 10%
RelativeWeight
Business RiskContribution
Securities TradingCapture Order
Client Side Validation
Market Side Validation
Check EligibilityCheck SuitabilityCheck Availability
Rectify OrderCancel Order
A natural consequence
© 2016 by
80%
26.7%
23.7%
1.5%
1.5%
53.3%
10%
10%
Business RiskContribution
Securities TradingCapture Order
Client Side Validation
Market Side Validation
Check EligibilityCheck SuitabilityCheck Availability
Rectify OrderCancel Order
39 12 18 31
60 2 6 32
33 17 50
100
42 21 37
34 5016
1025 25 40
18 22 3030
41 6 9 45
Business RiskCoverage %
The peak of perfection!
© 2016 by
LOWRISK
test
cas
es
N-2 N-1 N N+1 N+2 sprint
critical limit
90%
busi
nes
s ri
sk
75%20%HIGH
RISK
MEDIUM
RISK
…the time needed for testing is infinitely larger than the time available – C. Kaner.
Practical significance
© 2016 by
Busi
ness
Ris
k Co
vera
ge
60%
80%
100%
40%
20%
0% 0
Def
ect
Rate
max
100%
Test Cases20%0% 40% 60% 80%
Critical Defects
𝟏 Weight𝟏𝟏𝟏 𝟏𝟏𝟏… …
A value neutral approach to testing
© 2016 by
Randomness rules!No risk assessed, critical defects are just chance hits.
No characteristics!No plan about what drives the test, selective testing is impossible.
Nothing to learn!No chance to improve anyhow.
A value neutral approach to testing
© 2016 by
60%
80%
100%
40%
20%
0%Busi
ness
Ris
k Co
vera
ge
100%20%0% 40% 60% 80%
Test Cases
Critical Defects0
Def
ect
Rate
max
𝒘𝟏𝒘𝟐𝒘𝟑𝒘 𝟒 𝒘 𝒊 𝒘𝑵… … … … 𝒘 𝒊≥𝒘 𝒊+𝟏Weight
The first steps towards risk-based testing
© 2016 by
Know what matters most!Good knowledge about high-risk areas.
Know what doesn’t matter!Good knowledge about low-risk areas.
Still a lot to learn!Where to reassess risk? Consider defect density!
The first steps towards risk-based testing
© 2016 by
Scrutinize, re-evaluate & improve your risk-based approach
continuously.
*Strive after continuous improvement at defined checkpoints, instead of ultimate perfection. Your product is an ever changing entity, and so do
your risks?
D‘oh, risk-based testing contains risks!
© 2016 by
60%
80%
100%
40%
20%
0%
Busin
ess R
isk C
over
age
100%20%0% 40% 60% 80%
Test Cases
critical defects0
Def
ect
Rate
max
𝒘𝟏𝒘𝟐𝒘𝟑𝒘 𝟒 𝒘 𝒊 𝒘𝑵… … … … 𝒘 𝒊≥𝒘 𝒊+𝟏Weight
HighRisk
do a lot with a little
LowRisk
Testing everything without testing everything
© 2016 by
Doing a lot with a little!Achieve maximum risk coverage by minimum amount of test cases.
Be on the safe side!Learn continuously to keep the critical areas up to date.
Work as a brand protector!Remove as much as possible risk earliest possible.
Testing everything without testing everything
© 2016 by 22
Thank You for your attention
Host: Elmar PauwelsEmail: [email protected]
Follow us on Twitter: @tricentis
Stay in touch!