“the future of software testing” structured or agile, … › wp...structured or agile, the best...
TRANSCRIPT
![Page 1: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/1.jpg)
Structured or Agile, The Best of Both
WorldsDerk-Jan de Grood,
Collis, The Netherlands
Europe’s Premier Software Testing EventWorld Forum Convention Centre, The Hague, Netherlands
WWW.QUALTECHCONFERENCES.COM
“The Future of Software Testing”
![Page 2: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/2.jpg)
Leiden Singapore Minneapolis Edinburgh
Best of both worlds
By: Derk-Jan de Grood
Date: November 2008
Location: EuroSTAR
Exploratory testing in a structured way
![Page 3: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/3.jpg)
www.collis.nl 3
Objectives for this presentation
Some basic principles of exploratory and structured testing
Insight in the way different approaches can benefit from each other
Answer to the question „Why should I ?‟
![Page 4: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/4.jpg)
Mail exchange
DJ: I got the idea, also during discussion with John Bach, that in the US a lot of Exploratory testing is done as a separate thing. Thus, although structured it is not embedded in a test method. Do you disagree?
Lee: Depends on the organization –some see it as two different approaches (even done by different people). With others ET is very integrated. It‟s difficult to generalize.
www.collis.nl 4
![Page 5: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/5.jpg)
Regional differences ?
www.collis.nl 5
Administrative systems
Procedural testingProducts
Technical testing
![Page 6: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/6.jpg)
6
Schools of testing
Analytic School
Aka Structural testing
Code coverage
Verification
Standard School
Sees testing as a way to measure progress with emphasison cost and repeatable standards
Make sure each requirement is tested
Quality School
Emphasizes process and acts as Gatekeeper
Protect users from bad software
Testing is a stepping stone to process improvement
Context-Driven School
Emphasizes people,seeking bugs that stakeholders care about
Exploratory testing
Commercial market driven software
Agile School
Uses testing to prove that development is complete; emphasizes automated testing
Test Driven development
[Pettichord 2007]
![Page 7: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/7.jpg)
7
Which school would you apply ?
Which school would apply for testing :
Auto-pilot system for airplane (analytical)
SAP system (standard)
Website supporting a campaign (context)
Innovative system (agile)
Mobile phone (standard?)
Off-shore build system (quality?)
No specs available (context or agile)
Game (context)
![Page 8: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/8.jpg)
Result-driventesting
8
Schools of testing
Analytic Standard
Quality
Context
Agile
[Pettichord 2007]
![Page 9: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/9.jpg)
9
N-dimensions
Maturity
Organization
Formality
Aim of testing
Technology
Development
method
Organisation type
Testteam
(skills)Key risk area’s
(important quality attributes)
Time to
market
![Page 10: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/10.jpg)
3 approaches for testing
Unstructured
Scripted
Exploratory
www.collis.nl 10
![Page 11: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/11.jpg)
www.collis.nl 11
Exploratory Testing
![Page 12: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/12.jpg)
Scripted vs. Exploratory
www.collis.nl 12
100 % Exploratory 100 % Scripted
![Page 13: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/13.jpg)
13
What is Exploratory Testing?
An approach for unscripted testing based upon skills and experience of the tester. ET is a risk based technique using a formal procedure, test charters and heuristics.
The no 1. excuse for not having to prepare our test design in full detail: We do exploratory testing !
“Exploratory testing is simultaneous learning, test design, and test execution.”James Bach
![Page 14: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/14.jpg)
Real live situation
A Colleague said:
“The specs are outdated and incomplete. There is a need for manual regression scripts. In preparation phase our test scripts only contains the basic info (test purposes). During execution we write down the details”
www.collis.nl 14
![Page 15: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/15.jpg)
www.collis.nl 15
A test design technique
![Page 16: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/16.jpg)
16
Traditional techniques
Error in the s/wCoverage of the testdesign
Error that remains undiscovered
Error found with used techniques
![Page 17: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/17.jpg)
17
ET
Error in the s/wCoverage of the testdesign
1. Points of Interest (POI)
2. First tests executed
3. Plan next step basedupon test results
4. Define new POI
5. Cont. with next POI
6. Conclusion thus finished !
![Page 18: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/18.jpg)
Process and charters
www.collis.nl 18
Assign Charters
Preparation Test execution Session evaluation
Debriefing
Test Charters
Define Charters
![Page 19: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/19.jpg)
www.collis.nl 19
Best of both worlds
![Page 20: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/20.jpg)
20
Test plan
Result
General
Test
strategy
Test
estimate &
planning
Test Risk
Analysis Detail Test plan
(test project
specific test
strategy)
![Page 21: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/21.jpg)
Test strategy
www.collis.nl 21
Which information is
available?
cWhat kind of error
is expected ?
What is the impact
when this error occurs
while operational ?
Select a technique
that covers the
right domain
Select a technique
with right test
depth
TRA
Test basis
Test technique
Select technique
for which
information is
available
![Page 22: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/22.jpg)
22
Building our test design
Syntax
PCT
BVA
EP
Exploratory
“the puzzle changes the puzzling.”James Bach
![Page 23: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/23.jpg)
23
ET fits y/n ?
Y Rapid feedback
Extension to scripted tests
Find most important bug in shortest time
Check work of other tester
Little or no specifications
Domain knowledge available
N Feedback loop breaks down
Detailed calculations
(life) Critical functions
Testing of reliability/performance
Test ware is important
Testers are less skilled
Sourc
e:
ISEB P
ractitioner
–Im
pro
ve Q
S
![Page 24: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/24.jpg)
Test Risk Analysis
www.collis.nl 24
![Page 25: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/25.jpg)
Traceability
www.collis.nl 25
![Page 26: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/26.jpg)
Registration
www.collis.nl 26
Test charter 123
![Page 27: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/27.jpg)
Test reporting
www.collis.nl 27
![Page 28: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/28.jpg)
Befit based reporting
www.collis.nl 28
![Page 29: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/29.jpg)
Progress reporting
www.collis.nl 29
![Page 30: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/30.jpg)
www.collis.nl 30
Why should I ?
![Page 31: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/31.jpg)
31
Evaluation
The ET session gave us clear understanding of the quality of the system. This was achieved in a very short period.
The fun about ET is that its fundamentals are easily understood.
Jaap Azier (KPN)
In order to use ET effectively we need to take the lessons learned into account. In special the logging and scenario testing.
Still I am glad we did ET. It enabled other people to get insight in the quality of the system.
Carin Smits (KPN)
Exploratory testing is testing on the edge.ET means taking the most out of people, this implies your dealing with people issues.It is exciting to find the edge of „we have tested all the essential‟.
Hugo Achthoven Implementation Manager (KPN)
The project went well, great team working. Together we worked towards the best working method. In the end, we certainly have found it. This resulted in clear and traceable test results.
Jasper OvergaauwTestexpert (Collis)
Close collaboration between users and test expert provided a judgement on the quality of the system, in a only short term.
Matthijs JorissenTestexpert (Sogeti)
![Page 32: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/32.jpg)
Focus on result
www.collis.nl 32
![Page 33: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/33.jpg)
Exploratory Testing
A nice extension of your tool-box
Don‟t let your message get blurred
www.collis.nl 33
![Page 34: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/34.jpg)
www.collis.nl 34
Wrap-up
![Page 35: “The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software](https://reader034.vdocuments.site/reader034/viewer/2022042408/5f247d4ae36503219361808f/html5/thumbnails/35.jpg)
www.collis.nl 35
Questions?
Derk-Jan de [email protected]
CollisDe Heijderweg 1 2314 XZ LeidenThe Netherlands
www.testgoal.com
Leiden Singapore Minneapolis Edinburgh