advancing testing using axioms
TRANSCRIPT
Exploring v Testing:The New Paradigm for Testing
@paul_gerrard
Paul [email protected]
gerrardconsulting.com
Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS– Want to be agile rather than follow Agile
dogma– Have a pragmatic approach and are focused on
delivery– Want a solution that fits, not a badly fitting
suit.
Typical tester job spec?• Skills & Requirements
– Thorough C# knowledge including Linq– Good .Net experience including charting and winforms
applications.– Experience building and maintaining automated test
systems– Ability to interact with the developers to anticipate future
needs– Ability to influence to ensure good test procedures are
developed and implemented• Desirable Skills
– Some knowledge of Java for mobile testing– An interest in photography. Wearable Computing
(Internet o
f Eve
rything)
Agenda• Innovations that Drive Change• 3 Patterns of Software Development?• A New Model for Testing• Some Consequences• Close
I'll talk about models quite a lot, but not 'Model Based Testing'
What's happening?• Stampede to mobile computing -
Digital• Big Data• Internet of (Every)Thing, Pervasive
Computing• Continuous Delivery, DevOps• (Test) Analytics, Data-Driven
Development• Shift-left, embedded testers, no test
team• Did you really think Agile was the last
word?
Three Innovations That Are Driving
ChangeBeing agileShift LeftAnalytics
(IoT/IoE and others too, of course)
#1Agile doesn’t work
but being agile might
Agile is no longer "innovative"Challenging Agile and being
agile might beIs continuous delivery for
you?
Have you been Agiled?
#2Shift Left
Developers in test?Testers in development?
Automation, analytics, collaboration, experimentation
#3Test Analytics
Shift-Right?Analytics from production
Analytics from testAn opportunity for testing
Where is this leading?
Your guess is as good as mineBut here's my guess anyway
The Pressure to Change
The range of development approaches
Waterfall/StructuredAgile
‘Pure’ Agile
High Structure/Formality
• A one-dimensional view of the range of approaches
• Varying formality, but ALL require discipline
• But this one-dimensional view is too simplistic• Emerging out of the Agile approach are (what
I call) factory approaches• Lean, Kanban, Continuous Delivery blah
blah…
No man’s land?
Process Formality
Wikipedia – “Agile Methods”
Intelligent Definition and Assurance Slide 13
But…
There are three patterns, not just two• Many ‘new’ approaches are labelled 'Agile'
but– They aren’t Agile, they just work– Aimed at factory workers, not knowledge
workers• Continuous-delivery approach is appropriate
for rapid-delivery into online/mobile systems• CD advocates often have an Agile
background, so you might think CD is Agile• Not so.
Intelligent Definition and Assurance Slide 15
Three Software Development
Patterns?A Model
Three development patterns
Intelligent Definition and Assurance Slide 17
Structured
Agile Continuous
Goal-Bas
edHi-Process
Autonomous
Key pattern characteristicsCharacteristic SummaryStructure Structure of the project team
Pace/cadence What drives the rate of decisions making?
Leadership How is the team managed/directed?
Definition In what format is requirements knowledge captured?
Testing How is testing (mostly) performed?
Auto. Test When are automated tests created? Who leads?
Measurement What/how is project measurement performed?
Governance What form does governance take?
Intelligent Definition and Assurance Slide 18
All models are WRONG, some are USEFULNeedless to say, I’ll be generalising quite a lot
Profiles of the three patternsCharacteristic Structured Agile ContinuousStructure Managed team Autonomous Production Cell
Pace/cadence Business decision Team decision Feedback
Leadership Project Managed Guided Research Line Managed
Definition Fixed spec Dynamic spec Live Specs
Testing Scripted Exploratory Automated
Auto. Test Retrospective Developer led Pervasive
Measurement Pervasive Avoided Analytics
Governance Bureaucratic Trust-based Insight-Driven
Intelligent Definition and Assurance Slide 19
But there aren't three patterns
There are many
From high process to DIY process• In the past, high process was seen to be
the technical solution to the software problem
• But technical solutions cannot solve human problems
• Agile promised flexibility, but a lot of Agile is really 'high process-in-the-small'
• True agility means a team can adapt or select its process on-the-fly.
The old ways won't work in the future
We need a New Model of Testing (free from logistics)
Forget Logistics(for the time being)
Document or not?Automated or manual?
Agile v waterfall?This business or that
business?This technology v that
technology?
ALL Testing is Exploratory
We explore sources of knowledge ...
... to build test models ...... that inform our testing.
Examples of models
Models are innate, essential, human
Judgement, exploring and testing
Testing(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring (sources) Judgement
Creates testmodels
Uses testmodels
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers explore the same way? I think so.
Exploration process
ExplorationDefinitions specs/stories
People(& you)
Sources
Require-ments
TestModels
Enquiring
Challenging
Sources:People, documents,experience, system under test
Modelling
Test Models:Can be documented
or mental models
Predicting
System under test
Testing process
TestingSystem
Under Test
Refining
Informing Applying
Interpreting
TestModels
Revise the System
Logging Revis
ing
More exploring Reporting
New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
Some Consequences
There are others
Relation toTDD and BDD?
TDD is not really testing BDD is modelling using stories
Test automation from a different
perspectiveAutomation efforts fail too oftenAutomation uses different test
models
Capabilities
Enquiring, Modelling, Predicting, Challenging
Informing, Applying, Interpreting, Refining
Reporting and Logging
• Analysis, enquiry and elicitation• Modelling• Creation of custom models, using
heuristics, guesses, brainstorming, ideation, creative thinking
• Custom test design techniques• Comparison of models, value,
advantages, disadvantages, compromises
• Identification, validation and use of oracles
• Predicate logic and proof• Hypothesis and inference• Socratic method• Rapid Review and Inspection
techniques• Test case design• Test models and the meaning of
coverage• Testing as controlled experiment• Observation, Note taking,
recording
A very different skillset• Basic data analysis and statistics• Decision-making with
incomplete data• Computer forensics• Fault tree analysis• Failure diagnosis• Bug advocacy, triage processes
and negotiation• Meaningful software and test
metrics• Visual presentation of data• Reporting and presentation skills• Understanding stakeholders• Test analytics• Risk management, risk-based
testing and decision-making• Critical Thinking• Interpersonal skills• Dealing with
uncertainty/fallibility
Testing Career Development(speculative)
Foundations
Technical
Management
StrategicTest Strategy Project
IntelligenceTest Assurance
Exploration Forensics Interpretation
Scripting/Programming
Test Automation Technical (Excel, SQL, OS utils etc)
Stakeholder management
Analytics & visualisation
Managing uncertainty
Critical Thinking
ISTQB, TMMi etc...
Supplier Management
Test Process Management
Methodology
New Model Testing changes what
testers & developers need to
learnRoles of developers/testers become
blurredPerhaps they are not different disciplines
after all?Why have a tester certification?
Do Testers Need to Learn How to
Code?Some, perhaps many will
have to
Testers don’t own testing anymore
Be a TestMaster"Assurance through the
lifecycle"
The phase after development is REWORK, not
TestingTesting must align with
development, not compete with it or rescue it
Exploring v Testing:The New Paradigm for Testing
@paul_gerrard
Paul [email protected]
gerrardconsulting.com
Programme Chair 2014