derk jan de grood - et, best of both worlds
TRANSCRIPT
Leiden Singapore Minneapolis Edinburgh
Best of both worlds
By: Derk-Jan de Grood
Date: November 2008
Location: EuroSTAR
Exploratory testing in a structured way
www.collis.nl 2
Objectives for this presentation
Some basic principles of exploratory and structured testing
Insight in the way different approaches can benefit from each other
Answer to the question ‘Why should I ?’
Mail exchange
DJ: I got the idea, also during discussion with John Bach, that in the US a lot of Exploratory testing is done as a separate thing. Thus, although structured it is not embedded in a test method. Do you disagree?
Lee: Depends on the organization – some see it as two different approaches (even done by different people). With others ET is very integrated. It’s difficult to generalize.
www.collis.nl 3
Regional differences ?
www.collis.nl 4
Administrative systemsProcedural testing
ProductsTechnical testing
5
Schools of testingAnalytic School Aka Structural testing Code coverage Verification
Standard School Sees testing as a way to
measure progress with emphasison cost and repeatable standards
Make sure each requirement is tested
Quality School Emphasizes process and acts as
Gatekeeper Protect users from bad software Testing is a stepping stone to process
improvement
Context-Driven School Emphasizes people,
seeking bugs that stakeholders care about
Exploratory testing Commercial market driven
software
Agile School Uses testing to prove that
development is complete; emphasizes automated testing
Test Driven development
[Pettichord 2007]
6
Which school would you apply ?
Which school would apply for testing :
Auto-pilot system for airplane (analytical) SAP system (standard) Website supporting a campaign (context) Innovative system (agile) Mobile phone (standard?) Off-shore build system (quality?) No specs available (context or agile) Game (context)
Result-driventesting
7
Schools of testing
Analytic Standard
Quality
Context
Agile
[Pettichord 2007]
8
N-dimensions
3 approaches for testing
Unstructured Scripted Exploratory
www.collis.nl 9
www.collis.nl 10
Exploratory Testing
Scripted vs. Exploratory
www.collis.nl 11
100 % Exploratory 100 % Scripted
12
What is Exploratory Testing?
An approach for unscripted testing based upon skills and experience of the tester. ET is a risk based technique using a formal procedure, test charters and heuristics.
The no 1. excuse for not having to prepare our test design in full detail: We do exploratory testing !
“Exploratory testing is simultaneous learning, test design, and test execution.”James Bach
Real live situation
A Colleague said:“The specs are outdated and incomplete. There is a need for manual regression scripts. In preparation phase our test scripts only contains the basic info (test purposes). During execution we write down the details”
www.collis.nl 13
Exploratory testing or not?
www.collis.nl 14
A test design technique
15
Traditional techniques
Error in the s/wCoverage of the testdesign
Error that remains undiscovered
Error found with used techniques
16
ET
Error in the s/wCoverage of the testdesign
1. Points of Interest (POI)
2. First tests executed
3. Plan next step based upon test results
4. Define new POI
5. Cont. with next POI
6. Conclusion thus finished !
Process and charters
www.collis.nl 17
www.collis.nl 18
Best of both worlds
19
Test plan
Test strategy
www.collis.nl 20
21
Building our test design
Syntax
PCT
BVA
EP
Exploratory
“the puzzle changes the puzzling.”James Bach
22
ET fits y/n ?
Y Rapid feedback
Extension to scripted tests
Find most important bug in shortest time
Check work of other tester
Little or no specifications
Domain knowledge available
N Feedback loop breaks down
Detailed calculations
(life) Critical functions
Testing of reliability/performance
Test ware is important
Testers are less skilled
Sourc
e: I
SEB P
ract
itio
ner
– I
mpro
ve Q
S
Test Risk Analysis
www.collis.nl 23
Traceability
www.collis.nl 24
Registration
www.collis.nl 25
Test charter 123
Test reporting
www.collis.nl 26
Befit based reporting
www.collis.nl 27
Progress reporting
www.collis.nl 28
www.collis.nl 29
Why should I ?
30
EvaluationThe ET session gave us clear understanding of the quality of the system. This was achieved in a very short period.
The fun about ET is that its fundamentals are easily understood.
Jaap Azier (KPN)
In order to use ET effectively we need to take the lessons learned into account. In special the logging and scenario testing.
Still I am glad we did ET. It enabled other people to get insight in the quality of the system.
Carin Smits (KPN)
Exploratory testing is testing on the edge.ET means taking the most out of people, this implies your dealing with people issues.It is exciting to find the edge of ‘we have tested all the essential’.
Hugo Achthoven Implementation Manager (KPN)
The project went well, great team working. Together we worked towards the best working method. In the end, we certainly have found it. This resulted in clear and traceable test results.
Jasper OvergaauwTestexpert (Collis)
Close collaboration between users and test expert provided a judgement on the quality of the system, in a only short term.
Matthijs JorissenTestexpert (Sogeti)
Focus on result
www.collis.nl 31
Exploratory Testing
A nice extension of your tool-box
Don’t let your message get blurred
www.collis.nl 32
www.collis.nl 33
Wrap-up
www.collis.nl 34
Questions?
Derk-Jan de [email protected]
CollisDe Heijderweg 1 2314 XZ LeidenThe Netherlands
www.testgoal.com
Leiden Singapore Minneapolis Edinburgh