R&D testingVenturing under the surface of testing

Ondrej Tulach, Innovation Week 2018 Get slides: tulon.cz/QA

Testing: what does it mean officially?

„Procedure of submitting a subject of test to such conditions or

operations as will lead to its proof or disproof or to its acceptance or

rejection” (Merriam-Webster)

„Process concerned with planning, preparation and evaluation of SW to

determine that it satisfies specified requirements; to demonstrate that it's

fit for purpose; and to detect defects.” (ISTQB)

Questions, more questions

„Subject to conditions”? What ones? How to find out?

„Satisfies requirements”? Which? Where to get them?

„Is fit for it‘s purpose”? How do you prove that?

„Find defects”? How many is enough? How to find them? Should we count them?

Testing is a process, but far more is required to properly test

What is testing for in R&D practice?

Making sure the product works as it should

– by running scenarios similar to what the customer would do

– by ensuring that customer‘s use of product doesn‘t leave bad impression

(it‘s not cumbersome, annoying...)

Making it possible to fix defects in the product

– by discovering errorenous implementation, bad assumptions and

unhandled problems

Increasing confidence in product

– by successfully trying enough scenarios that we know the products is

reasonably robust

Quality Assurance isn‘t just „testing”

QA is interdisciplinary field consisting of many activities:

1. Testing the actual execution of testing scenarios

2. Test automation tests run&evaluate themselves→regression

3. Test analysis systematic creation and selection of scenarios

4. Risk analysis determining probability and impact of failures

5. Review (static m.) critical feedback to SW in each lifecycle phase

6. Quality management adjusting processes and tools to ensure quality

„Testing” itself is just a tip of the iceberg

Testing maturity (=quality) levels

1) Verification testing– demonstrate it works (in at least 1 way)

2) Destructive testing– 1) + try to break it, directly and indirectly; test

sanitization of inputs, error messages, invalid data, ...

3) Lowering overall risks– 1) + 2) + use Test Analysis tools to increase scenario

coverage of product and environment

4) Preventive Quality Assurance– 1) + 2) + 3) + Integrity levels and FMEA etc.

What is needed for QA?

1) Ability to actually subject the system to tests

– Access to an existing System Under Test in testable state

– All the authorizations and clearances neccessary for testing

– Appropriate testing data

2) Tools

– for testing, test results management, and automation

– for identifying and/or managing risks

3) Clear criteria what is correct and incorrect (no ambiguity!)

– functional specifications

– independent correct results for cross-checking

4) QAs

– Testers with enough attention and dutifulness – or Test Automation

– Test Analytics, QA a Risk managers, Test Automators, Reviewers

5) Time

But we don‘t need...

...testing (above Testing Maturity Level 1)

– Really? Because all the users will always use the product only the way you expect, never „their own way”? Because they‘ll only ever use your data? And because outages of power or HW affecting your products never happen?

...test analysis and risk analysis

– Really? Because all the scenarios which could trigger a failure you‘ll just suck from your thumb, not

forgetting any important one? Because all assumptions of developers, conscious and sub-

conscious, were correct, and the resulting code super-robust?

...test automation

– Really? Because you‘ll never ever change once finished code? Your changes will never break

anything, hence you could do without regression tests?

Practice: how did and does look QA in R&D?

1) Debugging: „fixing until it works”

2) Waterfall: split the development to phases per skills

3) Contemporary competing methodologies

a) „V-model”: freeze of specs, test horizontally as well as vertically

b) „Agile/Iterative”: develop piece-present to customer-rework

1) is used by newbies until they burn themselves, 2) uses industry

wherever quality is required, 3) is avalanching through IT

How do Debugging, Waterfall, V-model and Agile look from the QA‘s

perspective?

R&D and QA in „testing-by-debugging”

Prehistoric era of IT development

No differentiation between developers and QA

No advanced QA methods, tools or metrics used

Cycle of writing code-execution-modifications

When the code finally works to the developer with his data/scenario, it‘s considered OK

Problem: what if anyone runs the code in a different way than dev assumed, handled and sanitized? With different data? Or an unexpected state of system happens?

R&D and QA in „Waterfall”

Developers, architects, testers, test analysts, reviewers are all distinct specializations, jobs, departments

Semi-formalized process, product moves through isolated phases of design, development and testing

Problem: all testing activities occur at the end of development cycle, when the QAs get their hands on final code and documentation

User requirements

• Business analysis

• Requirements

• Functional specifications

Software development

• SW Specifications

• SW design

• SW coding

Testing activities

• Test analysis

• Testing and validation

• Regression test suites creation

Waterfall

R&D and QA in „V-model”

QA and Dev specializati-ons are distinct professions, but collaborate

Formalized process, but

Each phase does critical review of previous && is verified by independent tests on same level

Analysis and creation of tests happens parralel with coding, from same specs, instead of at the very end

R&D in „Agile/Iterative”

„Agile” = manifesto of priorities: – people > processes

– working SW > documentation

– feedback > specifications

– responding to change > following plan

„Iterativne” = deliver spall pieces often

Developed product changes during development per customer‘s feedback

What does that mean for QA?

„Agile development”:When you‘re halfway through developing a locomotive and the customer realizes he actually wanted airplane, and you should do and test it

QA in „Agile/Iterative” – challanges & solutions

1. No time for Functional Spec in iteration/PI

– Solution: do TA instead of Specs; ambiguities are resolved with the PO = „TDD”

2. No time for Test Analysis in iteration/PI

– Solution: testing has phases 1) basic & 2) advanced; TA is done parralel with testing 1); 2) is done later when TA is finished

3. „Agility” → „no difference between QAs and Devs” = regression 40 yrs back before Waterfall; no Test Analyst or QA mgmt. role

– Solution: QAs in the team must take on all the specialist roles: Tst, TAn, Rev, Risk mgmt,TAut!

R&D example: where is CA test-maturity-wise?

Entry training „ASE“ gives employees QA skills between level 1 and 2

Advanced opt-in training „Advanced QA: Test Analyst“ gives employees QA skills between levels 3 and 4

CA offers employees ISTQB exam and certification, a world standard in Quality Assurance

Frequent mistakes

„Testing could/should ensure defect-less product”

– More possible states in the Solaris Server than molecules in universe!How do you cover that with tests? → impossible to fully test nontrivial SW

„Random testing could substitute targeted”

– won‘t cover „low probability-high impact“ defects

– won‘t test errors in chain-of-inputs, usually not even boundary values

„Testing is a low-qualification job” (anyone could test)

– Only testing on maturity level 0, not higher – and you do want higher

– Test Analyst or Test Automator require skills on-par with Developer

Questions and answers

Go on, ask...