new model testing: a new test process and tool
TRANSCRIPT
![Page 1: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/1.jpg)
New Model Testing:A New Test Process and Tool
@paul_gerrardPaul [email protected]
gerrardconsulting.comSlide 1Intelligent Definition and Assurance
![Page 2: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/2.jpg)
My Goals in this webinar• Share some R & D with you• Get you thinking about your test
process and opportunities to improve• Show you a brief demo of a tool
we’re working on• Invite collaboration.
Intelligent Definition and Assurance Slide 2
![Page 3: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/3.jpg)
Agenda (my journey)
Intelligent Definition and Assurance Slide 3
Test Process
Will Robots Replace Testers?
Tool Architecture
A Demonstration
![Page 4: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/4.jpg)
Testing Styles, Approaches“Structured/Waterfall/ Staged” Testing• Systematic• Transparent, Documented• Reviewable• Auditable• Repeatable, measurable• Automatable• Inflexible, not responsive• Obsolescent/inaccurate
documentation• Prone to biases, inattention• Outdated process• Expensive, Inefficient• Unimaginative, boring
“Exploratory” Testing• Agile (with a small ‘a’)• Improvised, imaginative• Flexible and responsive to change• Faster, cheaper• More effective• Personally enjoyable• Not repeatable• Not easily automated• Little or no documentation• Hard to manage• Hard to scale• Opaque• Not auditable, measurable.
Intelligent Definition and Assurance Slide 4Some of these are perceptions – which vary
![Page 5: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/5.jpg)
Different strokes for different folks• I’ve never believed that these styles are
‘opposites’, rather they differ in emphasis• Like the Agile manifesto, it’s more a set of
values/preferences that drive behaviour:– Differences in scale and timing– Planned v improvisational– Heavy v light documentation– Process and governance v agility and freedom– Etc. etc.
• Is there a fundamental thought process that underpins all testing?
Intelligent Definition and Assurance Slide 5
![Page 6: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/6.jpg)
Forget Logistics(for the time being)
Document or not?Automated or manual?
Agile v waterfall?This business or that
business?This technology v that
technology?
![Page 7: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/7.jpg)
ALL Testing is Exploratory
We explore sources of knowledge ...
... to build test models ...... that inform our testing.
![Page 8: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/8.jpg)
Judgement, exploring and testing
Testing(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring (sources) Judgement
Createsmodels
Usesmodels
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers think/explore the same way? I think so.
![Page 9: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/9.jpg)
New Model Testing
My BBC talk: http://www.bbc.co.uk/academy/technology/article/art20150522113029398 29 page paper: http://dev.sp.qa/download/newModel
![Page 10: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/10.jpg)
The New Model• Makes no assumption about logistics, context• It is not a process model with entry/exit
criteria, procedures, deliverables etc.• All models are wrong, but I believe this is
useful– An attempt to understand our thought processes– Our brains can work on several processes
simultaneously– Can help us better understand, information flows,
feedback and review processes, automation etc.– Focuses attention on (test) model-making.
Intelligent Definition and Assurance Slide 10
![Page 11: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/11.jpg)
An Obvious Question
Is it feasible to combine the best of structured and
exploratory testing and create a new test approach?
![Page 12: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/12.jpg)
Will Robots Replace Testers?
Some research
There is a paper at: https://tkbase.com/resources/viewResource/14
![Page 13: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/13.jpg)
A recent study*…• Over the next two decades, 47% of jobs in the
US may be under threat• It ranks 702 occupations in order of their
probability of computerisation– Telemarketers: 99% likely– Recreational therapists: 0.28% likely– Computer programmers: 48% likely
• Something significant is going on out there• If programmers have a 50/50 chance of being
replaced by robots, we should think seriously on how the same might happen to testers.
* “The future of employment: how susceptible are jobs to computerisation?”http://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf
Intelligent Definition and Assurance Slide 13
![Page 14: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/14.jpg)
Some systems-related occupations
Intelligent Definition and Assurance Slide 14
Occupation Rank (out of 702)
Probability of Computeris-ation
Computer and Information Research Scientists 69 1.5%
Network and Computer Systems Administrators 109 3.0%
Computer and Information Systems Managers 118 3.5%
Information Security Analysts, Web Developers, and Computer Network Architects
208 21%
Computer Occupations, All Other 212 22%Computer Programmers 293 48%Computer Support Specialists 359 65%Computer Operators 428 78%Inspectors, Testers, Sorters, Samplers and Weighers
670 98%
![Page 15: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/15.jpg)
Test Automation = Mechanical Tools
What we REALLY need are THINKING TOOLS
![Page 16: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/16.jpg)
The term Test Automation misleads• It misleads as a label because the
whole of testing cannot be automated
• The label is bad, but the scope of Test Automation is what I call ‘Applying’ in the New Model
Intelligent Definition and Assurance Slide 16
![Page 17: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/17.jpg)
Testers need Thinking Tools• There are ten testing activities in the New Model– Test automation tools only support one: ‘Applying’
• The remaining nine activities (information gathering, analysis, modelling, challenging, test design and so on) are not well supported
• All require some level of thinking and skills• Checking is possible when a system and its
purpose are well understood and trusted• Test automation tools are simple in principle…
… compared to the rest of the test process.
Intelligent Definition and Assurance Slide 17
![Page 18: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/18.jpg)
Four quadrant model of intelligent test tools
Ability to Investigate
Abili
ty to
Cap
ture
Kno
wle
dge
• Text editors, Screen Shots
Models, visualisations, relationships, transformations
• Note Takers
• Mind Maps
• UML/Case Tools
Control, imagination, discernm
ent, foresight
• Pencil and paper, sketching tools
Intelligent Definition and Assurance Slide 18
![Page 19: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/19.jpg)
Intelligent Definition and Assurance Slide 19
TERMINATORTESTERNot Yet!
![Page 20: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/20.jpg)
Cervaya™Tool Architecture
A nine month bot journey(But it’s been a twenty year
testing journey so far)
![Page 21: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/21.jpg)
Intelligent Definition and Assurance
The vision thing• I want a bot partner/pair that supports
exploratory testing• You know my view (model)
of testing already• Can we use the explore v test paradigm in a bot
that allows you to:– Explore, take notes and model– Record ideas, risks, tests, outcomes and bugs– Generate reports and documentation as a
consequence, rather than requiring you write them• Codename: Cervaya (cervaya.com is a holding page for now)
Slide 21
![Page 22: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/22.jpg)
Schematic
Intelligent Definition and Assurance Slide 22
Schema ServerSchema
Manager(web site)
Schema Repository
(web service)
Cervaya Bot ClientRobot Engine
Cervaya Server
Actions(web services)
Application Web Site
Speech Recognition
Interface
Command Line
Interface
Administer Schemas through the Web
DownloadSchema Perform Robot
Actions through services
Application Reporting, Monitoring, Control,
Management
Many App Servers – one (or many) per Schema
robotschemascervaya
![Page 23: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/23.jpg)
The target user is…• Not really a freestyle exploratory tester
working in a start-up or small product company• Testers working in:– Regulated, high-integrity, safety critical– High documentation, high accountability– Environments where testers are constrained by their
processes– Testers struggling to apply ‘trad’ methods in an
Agile, Digital, DevOps environment• Why can’t we dictate a detailed test plan for a
bot to document, analyse, visualise?
Intelligent Definition and Assurance Slide 23
![Page 24: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/24.jpg)
Bot Schema – State Model+
Slide 24Intelligent Definition and Assurance
![Page 25: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/25.jpg)
Simplified Schema
Intelligent Definition and Assurance Slide 25
Session
Notes
Observation
Navigation
Modelling
Home
Always start here – ready to commence the
sessionIn session – everything is done inside a session
Places, features, forms and fields comprise the
model
Notes and observations are related to places,
features, forms or fields
Notes can be ideas, concerns, risks, questions etc.
Test
Tests focus on forms and fields (but also end-to-
end scenarios)
![Page 26: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/26.jpg)
Demo
A partial prototype
![Page 27: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/27.jpg)
The location hierarchy• Explorers create maps (or should do)• The system map is a hierarchy of
locations as follows:• Application– Places
• Features– Forms
» Fields• All other content is located with respect
to some level in this hierarchy.Intelligent Definition and Assurance Slide 27
![Page 28: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/28.jpg)
Demo agenda• Cervaya– apps, versions and charters
• Speechbot• Thinkbot• Cervaya– Explorer– Session Status/Log– Creating and running tests– Viewing the model
• D3 and system maps/data visualisations.
![Page 29: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/29.jpg)
A New Test Process
![Page 30: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/30.jpg)
Exploration support• Frustration:
– testers are unimaginative, working by-rote– constant pressure to cut costs
• Productivity of exploratory test approaches is proven• Testers want to explore, but the need for control and
documentation constrains them• Testers needs tools that can capture plans and tester
activity in real-time• The next generation will be led by tools that support
the exploration of sources of knowledge.• These tools might use a “Surveying” metaphor.
Intelligent Definition and Assurance Slide 30
![Page 31: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/31.jpg)
Staged ProcessSystem Testing Re-Test
Big, up-front Test Planning
RegressionTestDevelopment
Automation
From Staged to Continuous
Scoping exploration sessions
Interactive Testing Sessions
Clarifications
Continuous Integration
ChangesRefinements
Continuous Dev, Test and Delivery
![Page 32: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/32.jpg)
Test Process• Cervaya can be used to create test plans– Initially based on documentation– Evolving plan based on chartered sessions
• Testing in the small – chartered testing• Testing in the large – Cervaya has the
building blocks for end to end tests• Could it generate the documentation
required in regulated industries? Why not?
Intelligent Definition and Assurance Slide 32
![Page 33: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/33.jpg)
Real-Time test management• The activity of the test team can be
seen in real-time• Testers can see each other’s activity
and liaise when necessary• Developers can also see progress
and requests for more information very quickly– E.g. selected queries, test failures could
be posted to Slack.Intelligent Definition and Assurance Slide 33
![Page 34: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/34.jpg)
From Staged to Continuous Testing• In Waterfall projects, tests work in stages
– Hi- then low-level test plan, test procedures, execution, re-testing, regression-testing
• If testers are shifted left (into the development teams) they can:– Model features, forms as they are created by the developers– The test plan at a low level emerges and is accurate as it
iterates over time– Longer/end to end tests – not yet available but will be soon
• Tests and test run histories are manages against versions (and platforms/environments soon)
• The aim is to export tests as soon as practical to be automated and provide a feedback mechanism from tools.
Intelligent Definition and Assurance Slide 34
![Page 35: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/35.jpg)
The best of both worlds?“Structured/Waterfall/ Staged” Testing• Systematic• Transparent,
Documented• Reviewable• Auditable• Repeatable,
measurable• Automatable
“Exploratory” Testing
• Agile (with a small ‘a’)• Improvised,
imaginative• Flexible and
responsive to change• Faster, cheaper• More effective• Personally enjoyable
Intelligent Definition and Assurance Slide 35
![Page 37: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/37.jpg)
Q & maybe some A
![Page 38: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/38.jpg)
A new test process?• The “tester as surveyor” affects the
relationship of testing to development• A new style of testing process emerges:
– No more documentation created in a knowledge vacuum
– Iterative, incremental knowledge acquisition and capture process closely aligned with the delivery of features
• Could this be an Agile test process at last?• At least: it fits the increasingly popular Continuous
Delivery, DevOps development approaches.
Intelligent Definition and Assurance Slide 38
![Page 39: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/39.jpg)
System Surveying• A System Survey incorporates features, and captures
the architecture of the system from a user perspective– Testers pair with developers and survey features– The knowledge required to build systems emerges over time– So does the design of the system– So should the models and documentation produced by
testers• Surveys that evolve the System Model/Map are shared• The basis of component and system tests is the System
Model itself• No need for extensive scripts or test procedures!• The information required for scripting is in the model.
Intelligent Definition and Assurance Slide 39
![Page 40: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/40.jpg)
A process that suits automation• Test process comprises a sequence of parallel actions
– Sequence: survey, model refinement then testing– Parallel: small subsets of functionality selected for surveys– These processes are both iterative and incremental as
learning proceeds• Scalable: if you survey it, you can test it• Automate: What you can survey and test, you can
probably automate• “Humans make the early maps; tools will follow
the trails we make.”• We don’t need Machine Learning to do this:
– Simple tools make suggestions that better inform and enrich exploration and testing.
Intelligent Definition and Assurance Slide 40
![Page 41: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/41.jpg)
All exploration and testing is driven by charters• Charters represent the low-level
plans for all sessions• Charters can be reused for multiple
sessions, if required
Intelligent Definition and Assurance Slide 41
![Page 42: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/42.jpg)
Services• Can Cervaya handle web services as
objects to test?• In principle yes – use your
imagination:–Map the form/field information to
service/fields– Service tests map directly to form tests
• But we need to create an option that doesn’t require you to use your imagination.Intelligent Definition and Assurance Slide 42
![Page 43: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/43.jpg)
Sessions and logging• Sessions can be as short or as long
as you like• Testers typically work in 60-120
minute bursts – 90 minutes is typical• Every action in a session is logged
against the session (locations selected, created, notes taken, test created and run)
• The activity of all testers is logged and can be scrutinised for audit purposes.
Intelligent Definition and Assurance Slide 43
![Page 44: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/44.jpg)
Running a Test• When you run a test
– You are presented with the test details (navigation, input values and expected outcomes)
– You can log a test status, outcomes if different from expected and an interpretation
– You can also log a screen shot or other file• The test run log record the detail above and tags
the run with the app version/build defined in the charter
• You can (if required) run the same test multiple times in the same session and against the same version.
Intelligent Definition and Assurance Slide 44
![Page 45: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/45.jpg)
Challenges, opportunities• Is it feasible to
generate a high or low level test plan from Surveyor?
• Environments/platforms
• Integrations– Chatbot – Slack etc.– Internal IM– Stories– Test automation a la
BDD – both ways
• Visualisations– Coverage– Reporting– Impact analysis
• Other benefits– Instant replays– Heat map, floor plan– Tester paths– Real time notifications– Real time test(er)
management
Intelligent Definition and Assurance Slide 45
![Page 46: New Model Testing: A New Test Process and Tool](https://reader036.vdocuments.site/reader036/viewer/2022062523/58714aa31a28ab55588b6775/html5/thumbnails/46.jpg)
New Model Testing:A New Test Process and Tool
@paul_gerrardPaul [email protected]
gerrardconsulting.comSlide 46Intelligent Definition and Assurance