ipa lentedagen 2006: testing1 ipa lentedagen 2006 testing for dummies judi romijn [email protected]...
Post on 20-Dec-2015
217 views
TRANSCRIPT
IPA Lentedagen 2006: Testing 1
IPA Lentedagen 2006
Testing for Dummies
Judi [email protected]
OAS, TU/e
IPA Lentedagen 2006: Testing 2
Outline
• Terminology: What is... error/bug/fault/failure/testing?
• Overview of the testing process– concept map– dimensions– topics of the Lentedagen presentations
IPA Lentedagen 2006: Testing 3
What is...
error/fault/bug: something wrong in software
failure: manifestation of an error
(observable in software behaviour)
something wrong in software behaviour(deviates from requirements)
requirements:for input i,give output 2*i3
(so 6 yields 432)
software:i=input(STDIN);i=double(i);i=power(i,3);output(STDOUT,i);
output (verbose):input: 6doubling input..computing power..output: 1728
errorfailure
IPA Lentedagen 2006: Testing 4
What is...
testing: by experiment,– find errors in software (Myers, 1979)– establish quality of software (Hetzel, 1988)
a succesful test: – finds at least one error– passes (software works correctly)
test-to-fail
test-to-pass
IPA Lentedagen 2006: Testing 5
What’s been said?• Dijkstra:
Testing can show the presence of bugs, but not the absence • Beizer:
1st law: (Pesticide paradox) Every method you use to prevent or find bugs leaves a residue of subtler bugs, for which other methods are needed
2nd law: Software complexity grows to the limits of our ability to manage it• Beizer:
Testers are not better at test design than programmers are at code design• Humphreys:
Coders introduce bugs at the rate of 4.2 defects per hour of programming. If you crack the whip and force people to move more quickly, things get even worse.
• ...Developing software & testing are truly difficult jobs!Let’s see what goes on in the testing process
IPA Lentedagen 2006: Testing 6
Concept map of the testing process
IPA Lentedagen 2006: Testing 7
Dimensions of software testing1. What is the surrounding software development process?
(v-model/agile, unit/system/user level, planning, documentation, ...) 2. What is tested?
• Software characteristics (design/code/binary, embedded?, language, ...)
• Requirements (functional/performance/reliability/..., behaviour/data oriented, precision)
3. Which tests?• Purpose (kind of coding errors, missing/additional requirements, development/regression)
• Technique (adequacy criterion: how to generate how many tests)
• Assumptions (limitations, simplifications, heuristics)
4. How to test? (manual/automated, platform, reproducable) 5. How are the results evaluated? (quality model, priorities, risks) 6. Who performs which task? (programmer, tester, user, third party)
• Test generation, implementation, execution, evaluation
IPA Lentedagen 2006: Testing 8
Dimensions + concept map
2
1
6
2
66
4
35
IPA Lentedagen 2006: Testing 9
1: Test process in software development
V-model:
implementationcode
detaileddesign
specification
requirementsacceptance
test
systemtest
integrationtest
unittest
IPA Lentedagen 2006: Testing 10
1: Test process in software development
Agile/spiral model:
IPA Lentedagen 2006: Testing 11
1: Test process in software development
Topics in the Lentedagen presentations:• Integration of testing in entire development
process with TTCN3– standardized language– different representation formats– architecture allowing for tool plugins
• Test process management for manufacturing systems (ASML)– integration approach– test strategy
IPA Lentedagen 2006: Testing 12
2: Software
• (phase) Unit vs. integrated system• (language)
imperative/object-oriented/hardware design/binary/…
• (interface) data-oriented/interactive/ embedded/distributed/…
IPA Lentedagen 2006: Testing 13
2: Requirements
• functional: – the behaviour of the system should be correct– requirements can be precise, but often are not
• non-functional:– performance, reliability, compatibility,
robustness (stress/volume/recovery), usability, ...
– requirements are possibly quantifiable, and always vague
IPA Lentedagen 2006: Testing 14
2: RequirementsTopics in the Lentedagen presentations:• models:
– process algebra, automaton, labelled transition system, Spec#
• coverage:– semantical:
• by formal argument (see test generation)• by estimating potential errors, assigning weights
– syntactical– risk-based (likelihood/impact)
IPA Lentedagen 2006: Testing 15
3: Test generation: purposeWhat errors to find?Related to software development phase:• unit phase
typical typos, functional mistakes• integration
interface errors• system/acceptance: errors w.r.t. requirements
– unimplemented required features‘software does not do all it should do’
– implemented non-required features‘software does things it should not do’
IPA Lentedagen 2006: Testing 16
3: Test generation: technique
data-based
structure-based
black box white boxerror seedingtypical errors
efficiency...
black box: we don’t have acces to the software to be testedwhite box: we have access to the software to be tested
Dimensions:
IPA Lentedagen 2006: Testing 17
3: Test generationAssumptions, limitations• single/multiple fault:
clustering/dependency of errors• perfect repair• heuristics:
– knowledge about usual programming mistakes– history of the software– pesticide paradox
• ...
IPA Lentedagen 2006: Testing 18
3: Test generationTopics in the Lentedagen presentations:• Mostly black box, based on behavioural requirements:
– process algebra, automaton, labelled transition system, Spec#
• Techniques:– assume model of software is possible– scientific basis: formal relation between requirements and model
of software
• Data values: constraint solving• Synchronous vs. asynchronous communication• Timing/hybrid aspects• On-the-fly generation
IPA Lentedagen 2006: Testing 19
4: Test implementation & execution• Implementation
– platform– batch?– inputs, outputs, coordination, ...
• Execution– actual duration– manual/interactive or automated– in parallel on several systems– reproducible?
IPA Lentedagen 2006: Testing 20
4: Test implementation & executionTopics in the Lentedagen presentations:• Intermediate language: TTCN3• Timing coordination• From abstract tests to concrete executable tests :
– Automatic refinement– Data parameter constraint solving
• On-the-fly: – automated, iterative
IPA Lentedagen 2006: Testing 21
5: Who performs which task• Software producer
– programmer– testing department
• Software consumer– end user– management
• Third party– testers hired externally– certification organization
IPA Lentedagen 2006: Testing 22
6: Result evaluation• Per test:
– pass/fail result– diagnostical output– which requirement was (not) met
• Statistical information:– coverage (program code, requirements, input domain, output domain)– progress of testing (#errors found per test-time unit: decreasing?)
• Decide to:– stop (satisfied)– create/run more tests (not yet enough confidence)– adjust software and/or requirements,
create/run more tests (errors to be repaired)
IPA Lentedagen 2006: Testing 23
6: Result evaluationTopics in the Lentedagen presentations:• Translate output back to abstract
requirements– possibly on-the-fly
• Statistical information: cumulative times at which failures were observed1. fit statistical curve2. quality judgement: X % of errors found3. predict how many errors left, how long to continue assumptions: total #errors, perfect repair, single fault
IPA Lentedagen 2006: Testing 24
Dimensions + concept map
2
1
6
2
66
4
35
IPA Lentedagen 2006: Testing 25
Hope this helps...
Enjoy the Lentedagen!