gerlof hoekstra - omg what have we done - eurostar 2013
DESCRIPTION
EuroSTAR Software Testing Conference 2013 presentation on "OMG What Have We Done" by Gerlof Hoekstra. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/TRANSCRIPT
Gerlof Hoekstra, Atos
OMG, What Have We Done !?
www.eurostarconferences.com
@esconfs#esconfs
Gerlof Hoekstra (NL) Started as software engineer Early 90’s: Testing!
Sr Test Consultant/-Manager at Atos
From software testing to large scale integration testing◦ Complex business processes◦ Many stakeholders◦ Many (software) suppliers
I had a dream…
Implemented standard test procedures
10 years later: everything still the same
Dream came true: but does this make me happy?
Assembling my E2E test team
Certified test professionals or ….?
What has happened?
Somewhere, something must have gone wrong….
The Test Plan Independent Testing Test Reporting & The Go / No Go Advice
Entry/exit criteria
Defect databases
Defect handling workflows
Test case specifications
Defect severity definitions
…
…
Raise a
defectnew Open Assigned
Accept
defectanalyse Y fixed Deployedfix defect
in progress
Deliver fix
to test
Retest
defectRetest OK Y Validated Closed
PostponedOn hold, postpone defect N
RejectedN Accept
rejectionY
N
The creation of the test plan involves the following activities:1. Establishing the assignment2. Understanding the assignment3. Determining the test basis4. Analysing the product risks5. Determining the test strategy6. Estimating the effort7. Determining the planning8. Allocating test units and test techniques9. Defining the test products10.Defining the organisation11.Defining the infrastructure12.Organising the management13.Determining the test project risks and countermeasures14.Feedback and consolidation of the plan
The creation of the test plan involves the following activities:1. Establishing the assignment2. Understanding the assignment3. Determining the test basis4. Analysing the product risks5. Determining the test strategy6. Estimating the effort7. Determining the planning8. Allocating test units and test techniques9. Defining the test products10.Defining the organisation11.Defining the infrastructure12.Organising the management13.Determining the test project risks and countermeasures14.Feedbackand consolidation of the plan
Now look at this !?!
Shall we strip the Test Plan to the bone and never write those big, boring and useless documents anymore?
Shall we strip the test planning process to the bone and neverdo it alone at the office but always in heavy interaction with our stakeholders?
“Testing requires skills that developers generally don’t have”
“Testing should be done by independent professionals, specialists”
“You can’t let a butcher test his own meat”
Really??
Testing is part of our life …
“We strictly separate development and testing…”
“We pay the developing party a fixed price for the product they deliver…”
“We pay the testing provider a fixed price for each defect they discover…”
“But we subtract that from the fee for the developing party!”
Nail down entry/exit & acceptance criteria
Hire independent test supplier for acceptance testing, preferably fixed price based on:◦ function points◦ number of test cases◦ number of requirements◦ ---
Test supplier executes acceptance tests
The customer accepts
Is this the way to get a product accepted by its stakeholders?
Don’t we reduce testing to just a series of contractual checks?
Does this customer realize what he is (not) getting?
If we need to ‘dig somewhat deeper’: how about the contract?
Shouldn’t we (the test providers) at least warn our customer?
Numbers/Graphs/KPI’s
# test cases executed
# test cases passed
$ test cases failed
# defects found/fixed/open
Coverage %
Defect density
Defect Detection %
Lists of open defects
Traffic lights
GO/NO GO ADVICE
Does this mean:
we have a quality product now?
the product complies to the design?
we have a usable product?
We have reached 95% multiple condition
coverage
What on earth does that mean??
Must be good, 95% is almost everything
Multiple condition coverage
If my customer asks me (the test manager) for a go/no go advice, I consider that as a personal failure. I did not properly do my job!
If I did, what would my advice be worth anyway?
We covered 95% of the area
We found 60 mines
We disarmed 58 mines
The 2 mines we did not disarm are here
These are MSM MKII mines containing 4Kg of Composition B explosives
IS THIS USEFUL INFORMATION ?
WOULD YOU APPRECIATE MY GO/NO GO ADVISE?
Do you consider your ‘mature’ test process as a bless or a curse?
Shouldn’t I complain and just be glad with what we have accomplished?
Consider you own test plan template. If you would strip it to the bone:◦ What parts would you immediately throw out?◦ What should definitely be in there?
Consider your own test planning process. How could you make it◦ Lighter?◦ More effective?◦ More interactive?◦ Less boring?
To test suppliers◦ (Ethical question): Do you join the rat race of test commoditization/
should we warn customers for the consequences of excessive separation of testing? How can we?
To customers◦ Does your purchase department really know what they are doing when
they define all those fancy KPI’s.
◦ How can you build positive incentives into test SLA’s to stimulate testers to use their brains in stead of just following procedures and recipes?
Suggestions for alternative ways of reporting?
Can you imagine why I never use the traffic lights symbols in test reporting?
What would be a good alternative?