memes & fitness landscapes - analogies of testing with sci evol (2011)
TRANSCRIPT
Emerging Topics track
1
Memes andFitness Landscapesdown at the Schools-yard:Analogies of testing context withscientific evolution
Neil ThompsonThompson information Systems Consulting Ltd
©Thompson information Systems Consulting Ltd
v1.0
Slides 15 minutes (not “lightning”, but...),discussion 5 minutes (plus after, whenever!)
Emerging Topics track
2
©Thompson information Systems Consulting Ltd
Four, five, six... schools of software testing?(Updated version) March 2007Copyright © 2003-2007 Bret Pettichord. Permission to reproduce granted with attribution
Emphasis onadapting to the circumstances under which the product is developed and used
Annotations by Neil Thompson after the Bret Pettichord ppt (blue text), the list in Cem Kaner’s blog December 2006 (black text)
Emphasis on analytical methods for assessing the quality of the software, including improvement of testability byimproved precision of specifications and many types of modeling (Control):
Emphasis on standards and processes that enforce or rely heavily on standards
(Test-Driven):emphasis on code-focused testing by programmers
Oblivious /Groucho?
Emphasis on policing developers and acting as “gatekeeper”
Axiomatic?
Factory: Emphasis on reduction of testing tasks to routines that can be automated or delegated to cheap labour
, and other sources! (red text)
Neo-Holistic?(like C-D)
Emerging Topics track
3
Why I want to talk about this
• To understand better why and how different people work and communicate in different ways (eg balance of personal and situational factors)
• To try to broker more interactive and detailed debate (with less emotional confrontation?) between people exhibiting the characteristics of different Schools
• To help pave the way for the evolution of testing into the future, eg using power tools such as Grounded Theory with statistical analysis, Artificial Intelligence concepts such as Bayesian belief networks and genetic algorithms; testing AI itself!
© Thompson information Systems Consulting Ltd
Emerging Topics track
4
©Thompson information Systems Consulting Ltd
Precursor talk to this: “The Science of Software Testing”
System Requirements/Specification/ Design
Note: this is starting with some “traditional” views of testing & science
Test
“Aim to find bugs”
Testresult
Test result =“as aim”?
Experimentresult
Hypothesis
Experiment
Experiment result =
“as aimed”?
Test is“successful”
Falsificationconfirmed
Hypothesisnot yetfalsified
Test is so far“unsuccessful”Product
Part of thecosmos
“Aim tofalsify hypothesis”
Y
N
Y
N
Myers
Popper
Emerging Topics track
5
©Thompson information Systems Consulting Ltd
Other quality-relatedcriteria
This comparison informs the hot topic: testing versus “just” “checking”
System Requirements/Specification/ Design
“Check”
Expectedresult
Checkresult
Check result =
Expected?
Check“passes”
Check“fails”Product
System Requirements/Specification/ Design
Test
Wayscould fail
Testresult
Test result =appropriate
?
Quality-related info
Info onqualityissuesProduct
Other oracles
Y
N
Y
N
Sources: Michael Bolton blog and various Context-Driven material
Emerging Topics track
6
©Thompson information Systems Consulting Ltd
A complementary view: Testing as facilitating value flow through SDLC
LEVELS OF DOCUMENTATION,pushed by specifiers
WORKINGSOFTWARE
Accepted
System-tested
Integrated
Unit / Component-tested
FLOW OF FULLY-WORKING SOFTWARE, pulled by
customer demand
Requirements
+ FuncSpec
+ TechnicalDesign
+ Unit / Componentspecifications
+ Test Specifications
Emerging Topics track
7
©Thompson information Systems Consulting Ltd
Value Flow in SDLC as layerswith four characteristics
Levels of system &service integration
Levels of stakeholdersBusiness, Users,Business Analysts,Acceptance Testers
Architects,“independent” testers
Designers,integration testers
+ Businessprocesses
Developers,unit testers
Understanding of problem... Un
ders
tand
ing
of s
olut
ion.
..
Desired quality
Tested (“known”) quality
Levels of
spec-ificationRequirements
Functional &NF specifica-
tions
Technicalspec, Hi-level
design
Detaileddesigns
But, howmeasure &manage?
Remember: not only for waterfall or V-model SDLCs, rather iterative / incremental go down & up through layers of stakeholders, specifications & integrations
Emerging Topics track
8
©Thompson information Systems Consulting Ltd
Managing Value Flow through SDLC using VF ScoreCards
FinancialEfficiencyProductivityOn-time, in budget- Cost of quality
SupplierUpwardmanagement
Informationgathering
• Based on Kaplan & Norton Balanced Business Scorecard and other “quality” concepts
• Value chain ≈ Supply chain:– in the IS SDLC, each participant
should try to ‘manage their supplier’
– for example, development supplies testing(in trad lifecycles, at least!)
– we add supplier viewpoint to the other 5, giving a 6th view of quality
• So, each step in the value chain can manage its inputs, outputs and other stakeholders
CustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- Complaints
Improve-menteg TPI/TMM…PredictabilityLearningInnovation
ProcessComplianceeg ISO9000Repeatability
- Mistakes
ProductVERIFICATIONRisksTest coverage
- Faults- Failures
Emerging Topics track
9
©Thompson information Systems Consulting Ltd
Value Flow ScoreCards canbe cascaded
Pieces of a jig-saw puzzle!In addition to “measuring” qualityinformation within the SDLC:• can use to align SDLC principles with
higher-level principles from the organisation
Business Analysts Requirements Reviewers
Architects
Acceptance Test Analysts
Func Spec Reviewers
Designers
Acceptance Testers
Sys TestersSys Test Analysts
Tech Design Reviewers
Developers
Int Test Analysts
via pair programming?
Component Test Analysts, Designers & Executers?
AT Designers & Scripters
ST Designers & Scripters
IT Designers, Scripters & Executers
(...but don’t necessarily need all of these!)
Emerging Topics track
10
©Thompson information Systems Consulting Ltd
Using “Metaphysics” & Scienceas analogies to inform Value Flow
Levels of system &service integration
Levels of stakeholders
Business, Users,Business Analysts,Acceptance Testers
Architects,“independent” testers
Designers,integration testers
+ Businessprocesses
Developers,unit testers
Layers of quality
Static values:
• Inorganic
Dynam
ic values?
• Biological
• Social• Intellectual
Layers of science
......
......
...Sc
ale.
......
......
..Physics
Chemistry: InorganicChemistry: Organic
Biology (& systemsthinking)
Social sciences(Philosophy)
Emerging Topics track
11
What was that about layers of Science?
© Thompson information Systems Consulting Ltd Image from http://www.aaas.org/spp/dser/03_Areas/cosmos/perspectives/Essay_Primack_SNAKE.GIF
(Ouroboros: Greek Οὐροβόρος or οὐρηβόρος,
from οὐροβόρος ὄφις "tail-devouring snake”)
• There is a cascade (and approx symmetry!):– Biology depends on Organic Chemistry– Organic chemistry depends on the special properties of
Carbon– Chemical elements in the upper part of the periodic
table come from supernovae– Elements in the lower part of the periodic table come
from ordinary stars– Elements are formed from protons, neutrons, electrons
(Physics)– ... quarks... string theory?? etc
• It just so happens that humans are about equidistant in scale from the smallest things we can measure to the largest
• Then... humans have evolved to use tools, build societies, read, invent computers...
• So, it is possible to think of pan-scientific evolution as a flow of value
• Now, back to software lifecycles...
Sources: Daniel Dennett “Darwin’s Dangerous Idea” “cosmic Ouroboros” (Sheldon Glashow, Primack & Abrams, Rees etc)
Inventions by humans,
eg Social Sciences
Emerging Topics track
12
©Thompson information Systems Consulting Ltd
Invert that picture: Testing is reallyat the top, and should diverge fast!
• Each level of progress generates possibilities, which are tested
• Then, each level is a platform which, when established, is easily built uponby “cranes” (without having to worry about the details below)
• After the science levels...• humans made tools, talked and
co-operated• printing gave us another level • now, software is following
exponential growth• So, software testing should surf the
wave of evolution (not flounder in the shallows behind it)
• Kurzweil epochs
5: Bio methods integrated into technology?6: Intelligence into matter/energy patterns?
4: Technology
3: Brains2: Biology
1: Chemistry & Physics
+0: Maths?!
“SINGULARITY”
Emerging Topics track
13
©Thompson information Systems Consulting Ltd
Evolution: first, traditional Darwinian (ie biological)
Image from www.qwickstep.com
Emerging Topics track
14
©Thompson information Systems Consulting Ltd
Image from www.qwickstep.com
Biological reproduction & evolutionare controlled by Genes
Image from .schools.wikipedia.org
Diversity
Sophist-ication
Replication & Selection
Mutation
GENE
Emerging Topics track
15
©Thompson information Systems Consulting Ltd
Biological Evolution as Sophistication plotted against Diversity
Source: Daniel Dennett “Darwin’s Dangerous Idea”
Diversity
Sophistication
Emerging Topics track
16
©Thompson information Systems Consulting Ltd
But evolution is not smooth?
“Punctuated equilibra” idea originated by Niles Eldredge & Stephen Jay GouldImages from www.wikipedia.org
Sophistication
Diversity“Gradual” Darwinsim
Sophistication
DiversityPunctuated equilibria
“Explosion” in species, eg Cambrian
Spread into new niche,eg Mammals
Mass extinction, eg Dinosaurs
(equilibrium)
(equilibrium)
(equilibrium)
Sophistication
Diversity
Number ofspecies
Emerging Topics track
17
©Thompson information Systems Consulting Ltd
So... Evolution ofScience overall
Biology
Chemistry
Organic
Inorganic
Physics
Social sciences
• Arguably other sciences have not evolved smoothly either• Sudden advances, akin to punctuated equilibria in biological evolution
Emerging Topics track
18
©Thompson information Systems Consulting Ltd
OK, what’s this got to do with software testing?
Social sciencesTools
Language
Books
Computers
• We have an important and difficult job to do here!
• Social sciences evolution Tipping Points
(Malcolm Gladwell)
Emerging Topics track
19
©Thompson information Systems Consulting Ltd
...and computers are evolving, in both sophistication and diversity, faster thansoftware testing?
Computers1GL
ObjectOrientation
Internet,Mobile devices
ArtificialIntelligence?!
4GL
3GL
2GL
• Are we ready to test AI??
Emerging Topics track
20
©Thompson information Systems Consulting Ltd
How software testing has evolved so far?
Overall periods developed after Gelperin & Hetzel, “The Growth of Software Testing”, 1988 CACM 31 (6) as quoted on Wikipedia
PERIODDEBUGGING (Psychology)DEMONSTRATION (Method)DESTRUCTION (Art)
EVALUATION
PREVENTION (Craft?)
SCHOOL(S)
1957
1976
EXEMPLAR OBJECTIVES SCOPE APPROACH
Pre-
1983
1984
2000
Science?2011
Verify, +maybe Prove, Validate, “Certify”
Weinberg(1961 & 71) Test + Debug Programs Think, Iterate
Hetzel(1972)
Show meetsrequirements
Find bugsMyers(1976 & 79)
Programs
Programs, Sys,Acceptance
+ Walkthroughs, Reviews & Inspections
Kaner et al(1988 & 99)
Experiment &Evolve? Neo-Holistic?
Measurequality
?
Beizer(1984)
+ IntegrationFind bugs,show meetsrequirements,+prevent bugsFind bugs, in serviceof improving quality,for customer needs
Realistic, pragmatic,normal
Emerging Topics track
21
©Thompson information Systems Consulting Ltd
Can we retrofit the Schools to this history?
PERIODDEBUGGING (Psychology)DEMONSTRATION (Method)DESTRUCTION (Art)
EVALUATION
PREVENTION (Craft?)
SCHOOL(S)
1957
1976
EXEMPLAR OBJECTIVES SCOPE “SCHOOL”?
Pre-
1983
1984
2000
Science?2011
Weinberg(1961 & 71) Test + Debug Programs
Hetzel(1972)
Show meetsrequirements
Find bugsMyers(1976 & 79)
Programs
Programs, Sys, Acc
Kaner et al(1988 & 99)
Experiment &Evolve?
Neo-Holistic?
Measurequality
?
Beizer(1984)
+ IntFind bugs,show meetsrequirements,+prevent bugsFind bugs, in serviceof improving quality,for customer needs
Analytic
Standard(Control)
Quality
ContextDriven
“no schools, but...”
?
Agile(Test-Driven)
Factory
Emerging Topics track
22
©Thompson information Systems Consulting Ltd
The Philosophy of Science is also evolving!
Classical
Bayesianism,GroundedTheory...
Logical
• So, perhaps the Philosophy of Software Testing could learn from this, perhaps it’s also evolving?...
PositivismEmpiricism
PopperKuhn
LakatosLaudan
• Again, progress jerky, not smooth• Paradigm shifts akin to punctuated equilibria
Hull
Emerging Topics track
23
©Thompson information Systems Consulting Ltd
Memes as an extension of the Genes concept
Biological evolution
GENES
Theme developed from Daniel Dennett “Darwin’s Dangerous Idea”
Mental, social & cultural evolution
Platforms
Cranes
MEMES
Replication & Selection
Mutation
Ideas Beliefs PracticesSymbols
Gestures
Speech
Writing
Image from .www.salon.comTaxonomy from www.wikipedia.org
“Otherimitable
phenomena”
Rituals
Emerging Topics track
24
©Thompson information Systems Consulting Ltd
Some candidates for Memes in software testing
Always-considerEffectiveness
EfficiencyRisk management Quality management
Insurance Assurance
V-model: what testing against W-model: quality management
Risks: list & evaluate
Define & detect errors (UT,IT,ST)Give confidence (AT)
Prioritise tests based on risks
Tailor risks & priorities etc to factors
Refine test specifications progressively: Plan based on priorities & constraints Design flexible tests to fit Allow appropriate script format(s) Use synthetic + lifelike data
Allow & assess for coverage changes Document execution & management procedures
Distinguish problems from change requests Prioritise urgency & importance
Distinguish retesting from regression testing
Use handover & acceptance criteria
Define & measure test coverage
Measure progress & problem significance
Be pragmatic over quality targets
Quantify residual risks & confidence
Decide process targets & improve over time
Define & use metrics
Assess where errors originally made
Define & agree roles & responsibilities
Use appropriate skills mix
Use independent system & acceptance testers
Use appropriate techniques & patterns
Plan early, thenrehearse-run,acceptance tests
Use appropriate tools
Optimise efficiency
Source: Neil Thompson STAREast 2003 (not “best practices” but reference points for variation?)
Emerging Topics track
25
©Thompson information Systems Consulting Ltd
An example of a differentsoftware testing “memeplex”
Source: Neil Thompson BCS SIGiST 2002 review of Lessons Learned in Software Testing (Kaner, Bach & Pettichord)
Your career in software testing
The role of the tester
Thinking like
a tester
Testingtechniques
Bug
advocacy
Autom
atingtesting
Documenting testing
Interacting with programmers
Managing the testing project
Managing the testing group
Planning thetestingstrategy
Emerging Topics track
26
©Thompson information Systems Consulting Ltd
Memeplexes and Fitness Landscapes: fixed / flexible?
------- RUGGED &
FIXED? -------
Emphasis on adapting to circumstances under which the product is developed & usedContextDriven
FLEXIBLE?
Standard(Control)
standards and processes
Factorydelegated to cheap labour
reduction of tasks
routines can be automated
Agile(Test-Driven)
code-focused testing by programmers (needs
automation)
Analytic precision of specsanalytical methods
assessing quality
of software
testability
----- many types of modeling -----
Quality acting as “gatekeeper”
policing developers
FinancialEfficiencyProductivityOn-time, in budget
- Cost of quality
SupplierUpwardmanagementInfo from otherlevels of Treble-V model
CustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- Complaints
Improvement &Infrastructureeg TPI/TMM…PredictabilityLearningInnovation
ProcessComplianceeg ISO9000Repeatability
- Mistakes
ProductVERIFICATIONRisksTest coverage
- Faults- Failures
Emerging Topics track
27
©Thompson information Systems Consulting Ltd
Are these separate species which cannot interbreed?
(Control)
(Test-Driven)
Factory
• ... or is each part of an ecosystem with its suppliers and customers?
Emerging Topics track
28
©Thompson information Systems Consulting Ltd
Shouldn’t (or doesn’t) Context-Drivensubsume practices of other schools *when context is appropriate*?
(Control)
(Test-Driven)
Factory
Neo-Holistic?(or Context-Driven
itself?)
Emerging Topics track
29
©Thompson information Systems Consulting Ltd
Examples of how memeplexes can help ascend peaks of fitness landscapes
precision of specsanalytical methods
assessing quality
of software
testability
----- many types of modeling -----Analytic
FinancialSupplier Customer Improvement &Infrastructure
Process Product
ContextDriven Prioritise tests
based on risks
• Context-Driven thinking could address gaps and rebalance the scorecard; or...• It might prefer its own scorecard
PLUS... OR INSTEAD! Risks: list & evaluate
Quantify residual risks & confidence
Allow & assess for coverage changes
W-model: quality management
Measure progress & problem significance
Assess where errors originally made
Emerging Topics track
30
©Thompson information Systems Consulting Ltd
A Tester’s Taxonomy for Meme generation &transmission
Replication & Selection
MutationIdeas Beliefs PracticesSymbols
Gestures
Speech
Writing
Image from .www.salon.comTaxonomy from www.wikipedia.org
“Otherimitablephenomena”
Rituals
WEEKENDTESTING
V/W MODEL
IN BAR WITHWORKMATES
CONFERENCETALKS
WORKINGON APROJECT
BLOGS
BOOKS
COURSES
PAPERS
STANDUPMEETINGS
IN BED/PARK/BAR, THINKING
SPECIFICMETHODS & PROCESSES
STANDARDS
Emerging Topics track
31
©Thompson information Systems Consulting Ltd
Speculation on meme variations in software testing
Gestures
Speech
Writing
“Otherimitablephenomena”
Rituals
ContextDriven
AgileFactory
QualityAnalytic(Test-Driven)
Standard(Control)
Favoured Meme vehicles
School groups
PracticesSymbolsFavoured Meme types? IdeasBeliefs
BLOGS
COURSES (Certification)COURSES (Proficiency)
WEEKENDTESTING
BOOKS, PAPERS
STANDUPMEETINGS
V/W MODEL
CONFERENCE TALKS (but not necessarily the same conferences!)
STANDARDS
SPECIFIC METHODS& PROCESSES
Emerging Topics track
32
©Thompson information Systems Consulting Ltd
Not only Evolution, but Emergence: progress along order-chaos edge
Physics
Social sciences
Chemistry
Biology
• For best innovation & progress, need neither too much order nor too much chaos• “Adjacent Possible”• Might this also apply to testing?
ORDER
CHAO
S
ORDER
CHAO
S
ORDER
CHAO
S
ORDER
CHAO
S
Extrapolation from various sources, esp. Stuart Kauffmann, “The Origins of Order”, “Investigations”
Emerging Topics track
33
So... What’s the message, here?
• Regarding schools of software testing – I believe they are a very useful concept but:– please see analogies with species, fitness landscapes & ecosystems– don’t just preach to testers, educate the customers!– are schools really fixed around beliefs, or are they more flexible?
• When you think & communicate, try using the memes framework to better understand what you are building on, where you are innovating, and what you want to achieve, with what audience:– could “old school” people please blog and tweet more?– could Context-Driven people write more books please?– let’s share our experiences on projects
• Software testing is evolving, should continue to evolve:– future jumps could be quite big (“platforms & cranes”)– may be sudden paradigm shifts (cf punctuated equilibria,
Per Bak’s sandpiles)– most fruitful path is on the chaos-order boundary?
© Thompson information Systems Consulting Ltd
Emerging Topics track
34
Next steps already considering• Analyse more specific examples of memes in congenial & hostile
environments – the “Extended Phemotype”!• More analogies of testing with history & philosophy of science –
both for individual strategies and for improvement• Practical uses of Bayesianism to focus testing – already are some?
What can we actually use? What’s coming?• Is there correlation between personality (eg Myers-Briggs, Belbin)
and “membership” of schools of software testing?– Myers-Briggs fixed, Belbin can vary with situation & mood??
• Is the Cynefin construct of any use here? (pronounced “kanavin”?)– quadrants of systems/situations (simple, complicated, complex & chaotic)
may be suitable for different school-type behaviour – or, different responses of Context-Driven
– suggests steps to approach, which vary in usage & sequence: Sense, Categorise, Analyse, Probe, Respond, Just-act
© Thompson information Systems Consulting Ltd
Emerging Topics track
35
Questions to think about / discuss
• Are (some?) people “stuck” in a school because of:– their personalities, – upbringing, – education/inculcation, – deeply-held beliefs?...– the company they keep, or – the jobs they tend to get recruited for?– what their boss wanted yesterday?
• Are the schools themselves evolving – if so, how? Eg...– Context-Driven, from origins to book(s) to blogs to this conference?– Factory school in response to “maturing” of outsourcing / offshoring
market?– current Agile movements?
• Do some memes replicate in spite of not really helping their hosts?
© Thompson information Systems Consulting Ltd
Emerging Topics track
36
Questions to think about / discuss (continued)
• Shouldn’t Context-Driven, “by definition”, embrace practices of all the other schools *where appropriate* (or are other schools bad even in their own context? Are cultural / ethical divisions insoluble?)
• Analogies between testing schools and schisms in science, eg string theory v the others, frequentists v Bayesians
• How should individuals in software testing evolve?• How should software testing be preparing for the future, eg testing
Artificial Intelligence:– what happened to Genetic Algorithms?– what can we do with Grounded Theory?– and (as above) Bayesian?
• What other big innovations could be coming? (see Steven Johnson)
• Any memes about to die out?• Do we want to go beyond testing
only software?
© Thompson information Systems Consulting Ltd
Emerging Topics track
37
Main references & Acknowledgements– Bret Pettichord: “Four Schools” presentation– Dave Gelperin & Bill Hetzel paper, The Growth of Software Testing– testingreferences.com: testingtimeline– Stuart Reid paper, Lines of Innovation in Software Testing – Cem Kaner: blog, and “Software Testing as a Social Science” presentations– James Bach & Michael Bolton blogs, plus Kaner Bach & Pettichord “Lessons Learned...” book– Mike Smith: originating motivation & ideas, then co-development, of Value Flow ScoreCards – plus key
input from Isabel Evans – Robert Pirsig books: Zen & the art of Motorcycle Maintenance + Lila – Sheldon Glashow: cosmic Ouroboros– Charles Darwin books– Richard Dawkins: various books, esp. Extended Phenotype & Climbing Mount Improbable– Daniel Dennett: various books, esp. Darwin’s Dangerous Idea– Susan Blackmore, Robert Aunger, Kate Distin etc: various books on Memes– Matt Heusser, blog post 31 Jul 2009, esp. comments by Laurent Bossavit & James Christie– Peter Godfrey-Smith book: Theory and Reality (Philosophy of Science)– Stuart Kauffman: various books, eg Investigations– Ray Kurzweil book, The Singularity is Near– Jurgen Appelo book & website, Management 3.0– Sharon Bertsch McGrayne book: The Theory That Would Not Die (Bayes)– Dave Snowden, Cynefin
© Thompson information Systems Consulting Ltd
Emerging Topics track
38
©Thompson information Systems Consulting Ltd
• Thanks for listening! • Questions & discussion?