are you in control of testing, or does testing control you?
TRANSCRIPT
![Page 1: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/1.jpg)
Are you in control of Testing, or does Testing control you?
Mike SmithIndependent Consultant
![Page 2: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/2.jpg)
40th Year in IT!
![Page 3: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/3.jpg)
I’ve lived through a few technology eras!
![Page 4: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/4.jpg)
Getting the balance right can be difficult!
![Page 5: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/5.jpg)
Trying to stay relevant
• Tool Author – Test Process Management (T-Plan)• Businessman/Entrepreneur/Tester/Innovator• UK Representative to ISTQB• Chairman ISTQB Advanced Level Syllabus Working Group• Founder member of CAT (Certified Agile Tester) SIG• Practical Assessments (R&D Project)• Web TV & Digital Marketing in sport• Professional sports governance (in my spare time!)• Writing a book!
![Page 6: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/6.jpg)
Presentation Background
• Lack of control over projects/initiatives in IT, Business & Sport
• Common faults over this broad perspective• This presentation focuses on software testing as a form
of measurement• But it uses analogies and common characteristics across
other perspectives • I will use some examples and my experiences to help
highlight key points
![Page 7: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/7.jpg)
Presentation Background
Concerns over effectiveness of software testing as a measurement process
The subject of my book – Measurement & Testing!
![Page 8: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/8.jpg)
My theme for a book
“Effective Measurement and Testing Delivers Control of Anything”
![Page 9: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/9.jpg)
Measurement & TestingMy Axioms
• Testing is more than just Testing!• Software Testing can learn from other forms of
Measurement & Testing• Other forms of Measurement & Testing can learn
from Software Testing• Agile Development is helping the case for
Measurement & Testing• Software Testing took a wrong turn when it
became obsessed with Test Cases!
![Page 10: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/10.jpg)
Another Testing Axiom
•The answer to the ultimate question of Life, The Universe and Everything?
Answer: “42”(Hitchhikers Guide to the Galaxy – Douglas Adams)
•The answer to any question about Software Testing?
Answer: “It Depends!”(Dale Perry – SQE Inc. USA)
![Page 11: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/11.jpg)
3 Key Words
Control
Measurement
Testing (& Software Testing)
![Page 12: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/12.jpg)
Control – Some definitions
•The power to influence or direct people's behaviour or the course of events. (lead)
•Determine the behaviour or supervise the running of. (lag & lead)
![Page 13: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/13.jpg)
Control vs ManagementWhat’s the difference?
Corporate Governance Definition1.Strategic Direction of the Business
1.Effective Monitoring and Control of the Management of the Business
2.Accountable to the Shareholders of the Business
![Page 14: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/14.jpg)
Control vs Management?
![Page 15: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/15.jpg)
Being in Control – Can you answer these questions?
•What are my measures of success?•How will I know when I’ve met them?•How sure will I be that I’m going to meet them at any point in time?•How do I know where I am against them at any point in time?•Can I manage change effectively?•For an IT project, how useful are Test Cases to help me keep in control?
![Page 16: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/16.jpg)
Successful Outcomes?
• Reach destination port• Get there on schedule• All passengers & crew are safe• Passengers have not been (too) seasick!• Within budget (economical with fuel)• Obstacles avoided!
![Page 17: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/17.jpg)
How will I know when I’ve met the successful outcomes?
• What are the key measures & targets?• Speed?• Direction?• Position?• Quality Attributes?
![Page 18: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/18.jpg)
Keeping the ship under control!Key Drivers
Initiatives taken based these key goal & performance drivers
![Page 19: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/19.jpg)
Other measures to support the key drivers for Control
![Page 20: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/20.jpg)
There may be many management systems to support these measures
A complex measurement & management framework is needed
![Page 21: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/21.jpg)
Simple Hierarchical Model does not support the complex relationships
![Page 22: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/22.jpg)
What is Business Performance Measurement & Management?
• ‘Translating Strategy into Action’– Kaplan & Norton
• Use of Business Balanced Scorecard– Different perspectives
• Drives behaviour• Measures outcomes• Links actions to strategy• Leads to predictable outcomes
![Page 23: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/23.jpg)
Dangers of Performance Measurement & Management?
Measures that are badly constructed can destroy an organisation’s performance… effort may be misdirected into unproductive actions and a great deal of time and effort can be wasted.
‘Balanced Scorecard in a week – Mike Bourne, Pippa Bourne’
![Page 24: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/24.jpg)
5 Key principles
1. Generic application2. Objectives, Measures & Targets, Initiatives3. Separate “What” & “How”4. Cascading Scorecards
– One person’s ‘How’ is another person’s ‘What’– Measures & Targets become objectives for next
person 5. Lead & Lag Indicators
– Goal Indicators (reactive)– Performance Indicators (predicative)
![Page 25: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/25.jpg)
Scorecard Objectives, Measures, Targets & Initiatives
InitiativesObjectives TargetsMeasures
WHY WHAT HOW
![Page 26: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/26.jpg)
Scorecards linked bycascading ‘whats’ & ‘hows’
HOWWHAT
HOWWHAT
HOWWHAT
• Could be 1:many or
many:many
• Simplest way is 1:1
HOW
HOWWHAT
HOWWHAT
HOWWHAT
WHAT
![Page 27: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/27.jpg)
Summary of Outcome Driven Measurement Framework/Model
• Separate “What” & “How”• Complex Relationships in Model• Information Traceability matrix Top to
Bottom• Control of Complex Organisations &
Systems• Good fit with Agile Development
Practices!
![Page 28: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/28.jpg)
Now consider Software Testing Model
• What are the Measures & Targets that support the Software Testing Process?
• Test Cases?• Test Objectives?• Test Requirements?• Test Conditions?• Testers focus on Test Cases, but are they the
best Measures/Targets to drive behaviour?• Are they the best to manage risk and change?
![Page 29: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/29.jpg)
The problem with Test Cases
![Page 30: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/30.jpg)
Test(ing) & Software Testing
•The means by which the presence, quality, or genuineness of anything is determined; a means of trial (lag)
•Software testing is a process of executing a program or application with the intent of finding the software bugs. (lag) It can also be stated as the process of validating and verifying that a software program or application or product meets the business and technical requirements that guided it's design and development. (lag & lead)
•Testing is a risk reduction process (lead)
![Page 31: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/31.jpg)
The problem with Test Cases
Test caseA set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]
![Page 32: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/32.jpg)
The problem with Test Cases
• How big?• When can they be defined?• They may help me Test, but do they help me
Measure?• Whose Measures are they?• Do they separate the What from the How?• History of Test Cases (‘Bottom-up’)• What else fails if my Test Case fails - traceability?• Tool problems – wrong model• What if none defined?
![Page 33: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/33.jpg)
The problem with Test Cases
![Page 34: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/34.jpg)
1479 test cases: but are they “useful”? Now, let’sstart with a
classificationtree
1479 test cases, so
it must be good,right? Test
specificationprocess
Documentationto agreecoverage
![Page 35: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/35.jpg)
A Simple Hierarchical World
![Page 36: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/36.jpg)
A Complex Relational World!
![Page 37: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/37.jpg)
A simple world became a complex world!
![Page 38: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/38.jpg)
Test Model described by ‘methods’ people at large pan-European financial institution.
Test RequirementsRequirements
WHY
Test Design Specifications
Test Case Specifications
WHAT WHAT HOW
• “From Requirements to Test Case Specification”
in test management tool
![Page 39: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/39.jpg)
A complex model to support software testing
May or may notNeed complex
model:
IT DEPENDS!
![Page 40: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/40.jpg)
Guidelines for a better model
• Need to separate What & How• Model needs Complex Relationships – many:many• Test Conditions can be Considered targets for success,
organised by Measurement Attributes– Functional, Behavioural
• Can and should be defined early• Ability to cope with change• Know where you are at any point in time• Delivers Control of Complex Systems• Good fit with Agile Development Practices!• Business Outcome Driven Development
![Page 41: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/41.jpg)
What factors affect the level of measures and targets in a measurement framework?
• Risk• Size & Complexity• Dev Method• Test Level• Manageability & Maintainability• Tool usage• Ability to cope with change• Quality and level of artefacts• Regulatory requirements…....
![Page 42: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/42.jpg)
What factors affect the level of measures and targets in a measurement framework?
IT DEPENDS!!Setting good measures and targets is
difficult - it requires strong analytical skills
People often neglect the –ve measures and focus on the +ve
![Page 43: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/43.jpg)
What can the world of Measurement learn from Software Testing?
• Software Testing has been developed as a discipline for many years
• Very detailed processes and techniques compared with other forms of Measurement & Testing
• Software Test Techniques– Support Analysis and Design Activities
• Airport Example
![Page 44: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/44.jpg)
Other Experiences
• Making a new Financial Exchange Live
• An Office Move• ISTQB Syllabus Development• Practical Assessments• Governance in Sport• Mindset change and silos!
![Page 45: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/45.jpg)
WHAT’S MISSING?
General Model
Control
Management &Information
Systems
Measurement Framework
Measures &Targets
Objectives
Initiatives
![Page 46: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/46.jpg)
WHAT’S MISSING?
Testing Model
Control
Tests and Test Cases
Test Requirements/Test Conditions
Measures & TargetsSeparated from TestsMany:Many Reqs:Tests
Objectives/Requirements
Initiatives
![Page 47: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/47.jpg)
WHAT’S MISSING?
New Financial Exchange
New Market
Tests and Test Cases
Regulator Approval
SPECIFICATION
MeasurementFramework
RetrospectiveTest Conditions inComplex Model
![Page 48: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/48.jpg)
WHAT’S MISSING?
ISTQB Syllabi (Presented in Prague 2011)
Certificates(FL,AL,EL)
LearningObjectives &
Syllabus
VISION / STRATEGY
SPECIFICATION
Business Outcomes
REQUIREMENTS
![Page 49: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/49.jpg)
WHAT’S MISSING?
Sports Club Constitution Review
Need to Change
Old Subject Matter Experience
VISION / STRATEGY
CONSTITUTION/RULES
Terms of Reference
REQUIREMENTS
![Page 50: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/50.jpg)
My ‘Axioms’
“Test Driven Everything”
“Business Outcome Driven Development”
“Effective Measurement and Testing Delivers Control of Anything”
“Top Down, meets Bottom Up”
![Page 51: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/51.jpg)
References
• Testing Entities Paper – John Kent• IEEE Paper – ‘Keystone Testing Entity’ Mike
Smith & Neil Thompson• Value Flow Scorecards – Mike Smith & Neil
Thompson• Balanced Scorecard in a week – Mike & Pippa
Bourne• My book on Measurement & Testing – when?!
![Page 52: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/52.jpg)
What can you take away from this?
• If you are building a pure hierarchical model to manage testing, think again
• Can I optimise my model to give better control without it becoming unmanageable?
• Am I relying too much on my testing tools – do I need something else?
• Can I use my analytical skills to engage with the business better?
• I encourage you to think about these concepts, even if you think you are in control…...
![Page 53: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/53.jpg)
If you think you’ve found a good move,
look for a better one!
![Page 54: Are you in control of Testing, or does Testing control you?](https://reader035.vdocuments.site/reader035/viewer/2022062503/586fdc4b1a28ab18428b6425/html5/thumbnails/54.jpg)
My Measures of Success for this talk?!
1.Standing Ovation?2.Loud Applause?3.Polite Applause?4.Silence?5.I won’t be let into Russia again?!!